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Preface 


The  primary  reason  for  this  research  effort  was  that  it  was 
necessary  to  meet  the  requirements  for  a Master  of  Science  Degree 
in  Systems  Management.  Additionally,  my  interest  in  conducting 
research  on  the  Officer  Evaluation  System  (OES)  was  heightened  by 
the  fact  that  the  OES  has  a direct  effect  on  my,  as  well  as  every  other 
officer's,  future  in  the  Air  Force. 

This  research  effort  has  hopefully  provided  some  additional 
insights  on  one  aspect  of  the  complex  and  controversial  subject  of 
employee  appraisal:  the  evaluation  of  individual  potential. 

Several  individuals  have  made  this  research  effort  quite  a bit 
easier  and  deserve  my  most  sincere  expression  of  appreciation. 

First  of  all,  I wish  to  thank  Major  E.  J.  Dunne,  who  eased  my 
anxieties  about  thesis  work  greatly  by  providing  me  with  this  research 
topic  and  serving  as  my  advisor.  Thanks  are  also  in  order  for  the 
computer  help  provided  by  Major  Charles  W.  McNichols  and 
Captain  Harold  E.  Klick.  Finally,  a very  special  thank  you  is 
expressed  to  Molly  and  Bill  Bustard,  whose  courteous  and  helpful, 
attitudes  in  the  AFIT  library  anu  computer  center,  respectively, 
made  this  research  effort  much  more  bearable. 


Drew  W.  Browning 
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Abstract 


Many  organizations,  including  the  United  States  Air  Force, 
utilize  employee  appraisal  systems  which  require  the  evaluation  of 
indiviuval  potential  in  addition  to  the  appraisal  of  one's  past  perform- 
ance. With  regard  to  its  officer  evaluation  system,  che  Air  Force  has 
established  only  broad  guidelines  that  indicate  various  criteria  to  be 
used  in  the  evaluation  of  potential,  one  of  which  is  past  performance. 
Thus,  each  individual  officer,  either  as  a rater  or  as  a ratee,  makes 

ents  of  OER  ratings  based  on  his  or  her  perceptions  of  what 
criteria  should  be  used  to  evaluate  potential.  It  is  hypothesized  that 
if  widely  varied  perceptions  do  exist,  many  organizational  dvsfunc- 
Kons  i-nay  occur. 

This  research  effort  examines  Air  Force  officers'  perceptions, 
both  individually  and  in  groups,  of  what  is  involved  in  a rating  of 
potential.  In  order  to  determine  these  perceptions,  a judgment  model- 
ing, or  "policy  capturing"  research  approach  was  used.  A decision- 
making exercise  was  administered  to  381  active  duty  USAF  officers 
attending  Squadron  Officers'  School,  Air  Command  and  Staff  College, 
and  Air  War  College.  In  this  exercise,  officers  were  asked  to  judge 
the  appropriateness  of  OER  ratings  assigned  to  18  hypothetical 
officers,  based  on  different  levels  of  two  factors:  (1)  the  officer's 
past  performance,  as  illustrated  by  three  indicators  and  (2)  the 
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officer's  potential,  as  illustrated  by  three  indicators  other  Lhan  past 
performance.  Data  collected  from  the  exercise  were  then  analyzed 
to  test  various  hypotheses  concerning  the  relative  influence  of  per- 
formance and  (other)  potential  factors  on  officers'  judgments  of  OER 
ratings. 

The  results  of  the  research  indicate  that  Air  Force  officers  do 
not  view  the  OER  rating  as  being  solely  determined  by  past  perform- 
ance, although  the  performance  factor  was  weighted  most  heavily. 
Analysis  indicated  that  officers  used  different  judgment  policies  when 
considering  "good"  and  "bad"  OER  ratings  (potential  was  weighted 
more  heavily  when  judgments  were  made  of  "bad"  OER  ratings),  and 
that  they  were  generally  internally  consistent  in  txK:r  judgment 
policies.  Additionally,  several  different  analyses  indicated  that  the 
Air  War  College  students  used  judgment  policies  that  were  signifi- 
cantly different  than  the  other  groups;  specifically,  AWC  students 
generally  placed  greater  emphasis  on  potential  than  the  other  PME 
groups.  Furthermore,  the  AWC  group  displayed  both  the  lowest 
internal  judgment  consistency  and  the  lowest  group  R values  (an 
indication  of  the  degree  of  decision-making  homogeneity  within  the 
group). 


THE  PERFORMANCE  -POTENTIAL  DILEMMA: 


DOES  IT  EXIST? 

I Introduction 

This  research  effort  is  concerned  with  employee  appraisal 
systems --frequently  called  performance  appraisal,  merit  rating, 
personnel  evaluation,  and  a host  of  other  terms  that  are  all  related  to 
the  evaluation  of  individuals  regarding  their  work.  Although  subject 
matter  in  this  report  is  discussed  in  terms  of  the  U.  S.  Air  Force 
Officer  Evaluation  System  (OES),  many  of  the  inferences  drawn  from 
this  research  could  have  significant  implications  for  similar  civilian 
as  well  as  military  personnel  appraisal  systems. 

George  Terry,  a noted  authority  on  management,  has  stated  that 
as  supervisors,  we  are  constantly  judging  and  forming  opinions  about 
the  contributions  and  abilities  of  our  group  members.  The  question, 
therefore,  is  not  whether  to  appraise  employees,  but  how  to  appraise 
them  (Terry,  1974:224).  If  an  organization  does  not  dearly  address 
and  answer  this  question  of  how  to  appraise  employees,  then  a number 
of  problems  may  arise  and  render  the  appraisal  system  ineffective,  or 
only  marginally  effective,  in  accomplishing  its  stated  goals. 

The  subject  of  employee  appraisal  has  a reputation  of  generally 
being  looked  upon  with  disdain  by  managers,  even  though  appraisal 
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systems  are,  by  design,  an  aid  to  management  rather  than  a 
hindrance.  There  has  therefore  been  a great  deal  written  about  the 
problems  associated  with  appraisal  systems,  and  there  are  a number 
of  "solutions"  to  these  problems  presented  in  the  form  of  recom- 
mended types  of,  or  variations  to,  appraisal  systems. 

One  source  of  problem  associated  with  appraisal  systems  relates 
to  the  question  of  what  the  system  is  designed  to  evaluate,  and 
subsequently  how  well  this  "design"  corresponds  to  what  is  actually 
evaluated  in  practice.  Appraisal  systems  that  are  designed  to  evaluate 
and/or  predict  more  than  simple  job  performance  (for  example,  an 
estimation  of  an  individual's  future  worth  as  well  as  his  or  her  past 
performance)  may  result  in  differing  perceptions  among  raters  and 
ratees  regarding  what  is  actually  being  evaluated  and  what  degree  of 
importance  each  factor  has  in  determining  an  individual's  overall 
appraisal  rating.  One  such  appraisal  system  that  is  designed  to  evalu- 
ate more  than  job  performance  is  the  Air  Force  Officer  Evaluation 
System. 


Air  Force  Officer  Evaluation 

A widely  discussed  and  controversial  topic  among  Air  Force 
officers  today  is  the  officer  evaluation  system.  Even  though  changes 
in  the  evaluation  system  in  recent  years  (e.g,  forced  distribution  rat- 
ings and  the  recent  revision  of  distribution  quota  percentages)  have 
heightened  the  controversy  and  interest,  the  basic  fact  remains  that 
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any  system  which  affects  officers  as  much  as  the  officer  evaluation 
system  will  always  generate  a great  deal  of  personal  interest  and  con- 
cern. In  order  to  more  closely  address  the  question  of  why  officers 
are  so  intensely  concerned  -with  the  OES,  one  need  only  examine  the 
intended  objectives  of  the  system.  According  to  Air  Force  Regulation 
36-10,  Officer  Evaluations,  the  objectives  of  the  officer  evaluation 
system  are  as  follows: 

The  officer  evaluation  system  must  provide  the  Air 
Force  with  essential  information  for  use  in  personnel 
decisions,  such  as  promotions,  assignments,  and  scnool 
selections.  The  system  permits  better  identification  of 
officer  quality  differences,  and  assists  in  identifying  and 
motivating  officers  for  due  course  and  accelerated  pro- 
motions (AFR  36-10,  1978:1-1). 

Thus,  the  officer  evaluation  is  not  intended  to  be  a simple  perform- 
ance appraisal  apparatus  which  seeks  only  to  judge  an  officer's  past 
performance  relative  to  other  officers  being  rated:  an  officer's  future 
worth  is  also  "evaluated,  " again  relative  to  other  officers  being  rated. 
The  OES  leads  to  key  decisions  regarding  an  officer's  career  progres- 
sion and  is  therefore  a primary  factor  in  determining  his  or  her  future 
in  the  Air  Force. 

Two  related  but  distinct  concepts  which  form  the  basis  of  the 
Air  Force  officer  evaluation  system  are  those  of  performance  and 
potential.  The  Air  Force  Form  707,  "Officer  Effectiveness  Report,  " 
(OER),  has  two  evaluation  sections:  Part  III  involves  a rating  of  per- 
formance factors,  and  Part  IV  involves  an  evaluation  of  potential. 

This  topic  will  be  discussed  in  more  detail  in  chapter  three  of  this 


report,  along  with  a historical  development  of  the  current  OER  system. 
The  current  OER  form  for  company  grade  officers  is  located  in 
Appendix  B. 

Since  the  officer  effectiveness  report  (as  an  instrument  of  the 
officer  evaluation  system)  requires  an  assessment  of  both  performance 
and  potential,  one  might  argue  that  the  individuals  involved  with  the 
OES  should  be  able  to  clearly  distinguish  between  these  concepts  if  the 
appraisal  system  is  to  effectively  accomplish  its  desired  objectives  as 
outlined  in  pertinent  Air  Force  regulations.  It  should  be  pointed  out 
that  this  distinction  affects  every  Air  Force  officer,  either  as  a rater 
or  as  a ratee. 

Performance  can  be  defined  as  the  degree  to  which  an  employee 
has  satisfactorily  accomplished  his  or  her  job  or  assigned  duties  dur- 
ing a specified  period  of  time,  and  is  determined  primarily  by  one's 
abilities,  coupled  with  motivational  forces.  Measuring  performance 
may  be  thought  of  by  many  as  a simple  process  of  identifying  an  indi- 
vidual's units  of  work  output.  However,  many  jobs,  including  a Large 
proportion  of  high  level  managerial  and  professional  jobs,  simply 
have  no  readily  identifiable  units  of  output;  in  these  instances,  the 
appraiser  observes  the  employee's  performance  and  records  the 
observations  with  some  form  of  appraisal  instrument  (Cummings, 
1974:4). 

Potential,  on  the  other  hand,  is  a concept  which  is  related  to 
the  future.  Although  potential  is  not  always  predicted  in  appraisal 
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systems,  it  is  more  prevalent  in  appraisal  systems  for  management 
personnel.  For  the  purpose  of  the  officer  evaluation  system  in  the 
Air  Force,  potential  is  defined  in  AFR  36-10  as  follows: 

Potential  is  the  ratee's  capability,  relative  to  that  of 
officers  in  the  same  grade  in  the  group  being  evaluated, 
for  expanded/more  diverse  responsibility.  Potential  for 
the  purpose  of  CER  rating  decisions  will  be  determined 
primarily  by  the  evaluator's  assessment  of  the  ratee's 
accomplishments  during  the  period  of  the  report  with 
consideration  given  to  other  whole  person  factors,  e.g. 
career  pattern,  integrity,  PME,  etc.  (AFR  36-10,  1978: 

2-3). 

Since  potential  is  future  oriented,  measurement  of  this  individual  capa- 
bility can  be  a difficult  task.  Generally  speaking,  raters  will  examine 
past  performance,  trends  in  performance,  and  personality  traits  in 
order  to  estimate  future  performance.  AFP  36-26  indicates  that  joK 
performance  is  normally  the  single  most  accurate  indicator  of  potential 
(AFP  36-26,  1977:3-1).  It  is  noted,  however,  that  the  degree  to  which 
other  factors  determine  this  rating  of  potential  is  not  addressed  in  Air 
Force  policy. 

The  concepts  of  performance  and  potential  are  discussed  defini- 
tionally  in  Air  Force  regulations,  but  not  specifically  addressed  are 
the  inherent  problems  involved  with  a failure  by  raters  and/or  ratees 
to  clearly  distinguish  between  these  concepts  when  actually  making 
judgments  of  various  OER  ratings  (Section  IV  racing,  "Evaluation  of 
Potential").  More  specifically,  since  potential  appears  to  be  difficult 
to  assess  in  terms  of  various  traits,  abilities,  and  motivation,  there 
may  be  a natural  inclination  for  raters  to  place  a heavy  (if  not  total) 
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emphasis  of  performance  factors  when  judging  an  individual's 
potential. 

Previous  Research 

There  have  not  been  many  studies  conducted  that  have  examined 
the  distinction  between  the  concepts  of  performance  and  potential  with 
respect  to  judgments  of  appraisal  ratings.  The  relationship  between 
these  concepts  and  the  necessity  of  a clear  distinction  between  the  two 
was,  however,  specifically  addressed  in  one  study  of  the  officer  evalu- 
ation system: 

"The  basis  of  many  past  problems  with  military  officer 
evaluation  systems  is  that  seldom,  if  ever,  have  past  sys- 
tems clearly  distinguished  between  performance  and  poten- 
tial, i.  e.,  have  not  separated  appraisal  of  the  immediate 
past  performance  from  appraisal  of  future  performance. 
Performance  appraisal  and  potential  appraisal  are  different 
processes  which  use  different  input  information  and  whose 
results  should  have  different  uses.  " (Dunne,  1977:22,  321 

Additionally,  one  thesis  research  effort  which  examined  junior 
officer  perceptions  of  the  OER  system  resulted  in  the  author’s  con- 
clusion that  junior  officers  feel  as  though  job  performance  is  the  most 
important  factor  making  up  their  potential.  The  author  additionally 
pointed  out  that  whether  or  not  job  performance  can  be  the  prime 
measure  of  potential  is  open  for  debate  (Keyserling,  1976:63).  This 
research  effort  further  investigates  this  subject  area  by  attempting  to 
answer  the  question  of  whether  officers  distinguish  between  the  con- 
cepts of  performance  and  potential  wnen  making  judgments  of  OER 
ratings,  as  well  as  whether  or  not  officers  are  consistent  in  making 


Due  to  the  interrelated  nature  of  performance  and  potential,  as 
well  as  the  difficulties  involved  in  assessing  the  two  factors  in  an 
officer  evaluation  system,  there  is  an  ever-present  possibility  that  an 
individual's  concepts  of  performance  and  potential  could  become  con- 
fused and  overlap  significantly  when  actually  making  judgments  of  OER 
ratings.  It  should  be  noted  that  this  confusion  is  not  to  be  interpreted 
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the  two  will  be  limited  to  individual  perceptions.  These  varied  indi- 
vidual perceptions  may  subsequently  have  dysfunctional  effects  that 
could  undermine  the  intended  goals  of  the  officer  evaluation  system. 

A discussion  of  the  effects  of  such  perceptual  differences  is  found  in  a 
1977  article  titled  "An  Integrated  Approach  to  Control  System  Design" 
(Ansari,  1977).  Ansari  proposed  that  successful  control  requires 
leaders  who  can  create  conditions  which  motivate  workers  toward 


desired  goals;  and  subordinates  must  share,  or  at  least  must  not 
reject,  their  manager's  perceptions  of  what  the  goals  are,  how  they 
are  to  be  met,  and  how  well  they  were  met  (evaluation).  Ansari  makes 
the  following  statement  about  evaluation: 

"The  greatest  and  potentially  most  harmful  (perceptual) 
differences  are  likely  to  be  at  the  evaluation  stage.  In  fact, 
different  perceptions  of  means  and  ends  may  well  affect  a 
subordinate's  performance  and  thus  show  up  in  evaluation 
also.  " (Ansari,  1977:109) 


Approach  to  the  Problem 

This  research  effort  investigates  Air  Force  officers'  perceptions 
of  performance  and  potential  in  terms  of  the  influence  that  each  con- 
cept has  on  judgments  of  OER  ratings.  In  order  to  evaluate  these 
perceptions,  a policy  capturing  approach  was  used,  along  with  an 
additional  survey  measure  of  officers'  more  intuitive  reactions  to 
either  receiving  or  assigning  both  "good"  and  "bad"  OER  ratings.  The 
nature  and  use  of  policy  capturing  techniques  in  investigating  rater 
policies  will  be  discussed  in  a subsequent  chapter  of  this  report. 


8 


Objectives 

The  overall  objective  of  this  research  was  to  investigate  the 
relationships  between  officers'  concepts  of  performance  and  potential 
and  their  judgments  of  appraisal  (OER)  ratings. 

Specific  objectives  that  were  pursued  in  support  of  the  overall 
thesis  objective  were  as  follows: 

1.  Develop  a theoretical  background  for  the  research  subject  by  con- 
ducting a literature  search  on  employee  appraisal  systems,  with 
particular  emphasis  on  the  role  of  performance  and  potential 
assessment  in  fulfilling  the  goals  of  appraisal  systems. 

2.  Develop  and  administer  a decision-making  exercise  that  "captures" 
the  policies  of  Air  Force  officers  regarding  their  judgments  of  OER 
ratings,  based  on  performance  and  potential  criteria. 

3.  Analyse  the  data  collected  through  the  decision-making  exercise  in 
order  to  determine  the  relative  weights  that  officers  place  on  per- 
formance versus  (other)  potential  criteria  when  making  judgments 
of  OER  ratings. 

4.  Analyze  the  data  collected  through  the  decision-making  exercise  in 
order  to  determine  the  consistency  with  which  officers  make  judg- 
ments of  OER  ratings  based  on  performance  and  potential  criteria, 
both  individually  and  as  groups. 

5.  Examine  the  correlation  between  officers'  intuitive  reactions  to 
OER  ratings  (Part  III  of  the  exercise)  and  the  results  of  the  policy 
capturing  analysis  (Part  II). 
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The  data  obtained  in  the  research  was  used  to  investigate  the 

following  research  hypotheses: 

HI:  Air  Force  officers  combine  performance  and  potential 
criteria  in  an  additive  rather  than  interactive  manner  when 
making  judgments  of  OER  ratings.  (Note:  This  hypothesis 
facilitates  the  policy  capturing  data  analysis.) 

H2:  Air  Force  officers,  when  considering  (1)  job  factors  which 
are  indicators  of  immediate  past  performance  and  (2)  personal 
factors  which  are  (other)  indicators  of  potential,  will  make 
judgments  of  OER  ratings  (Section  IV,  ’’Evaluation  of  Potential") 
based  equally  on  *-he  job  and  personal  factors. 

H3:  Air  Force  officers  use  the  same  judgment  policies  when 
judging  either  "good"  or  "bad"  OER  ratings  (i.  e.  "1"  and  "3," 
respectively)  in  terms  of  the  performance  and  potential  factors 
that  influence  these  judgments. 

H4:  Air  Force  officers  are  internally  consistent  in  their  judg- 
ment policies  concerning  OER  scores. 

H5:  Air  Force  officers  represent  a homogeneous  group  of 
decision  makers  regarding  OER  scores;  i.  e.,  use  the  same 
decision  policies  when  making  judgments  of  OER  ratings,  regard- 
less of  data  groupings  by  PMF  school,  aeronautical  rating,  OER 
rating  history,  and  whether  or  not  officers  have  been  OER  raters. 

H6:  The  relative  weights  for  performance  and  potential  criteria 
determined  by  the  policy  capturing  model  are  consistent  with  the 
intuitive  responses  given  in  Part  III  of  the  exercise;  i.  e. , 
modeled  judgment  policies  are  consistent  with  officers'  intuitive 
reactions. 


Scope  and  Limitations 


This  study  involved  an  initial  search  of  the  literature  on  the 
subject  area  of  employee  appraisal  systems  and  a subsequent  investi- 
gation of  the  perceptions  of  Air  Force  officers  regarding  the  concepts 
of  performance  and  potential.  It  is  emphasized  that  this  research 
effort  was  not  intended  to  be  a critique  of  the  officer  evaluation 


system.  Additionally,  it  was  not  an  attempt  to  determine  what 


importance  should  be  associated  with  various  predictors  of  potential. 
Rather,  investigation  was  directed  at  empirically  determining  whether 
officers  distinguish  between  the  concepts  of  performance  and  potential 
when  making  judgments  of  OER  ratings  and  the  consistency  with  which 
officers  make  these  judgments  based  on  performance  and  potential 
criteria. 

Although  this  research  was  aimed  at  determining  the  judgment 
policies  of  Air  Force  officers  in  general,  only  a small  sample  could  be 
selected  to  participate  in  the  decision-making  exercise.  For  this 
reason  the  sample  population  was  restricted  to  students  at  the  three 
Professional  Military  Education  (PME)  schools  at  Maxwell  AFB, 
Alabama.  The  exercise  was  pretested  at  the  Air  Force  Institute  of 
Technology. 

One  limitation  associated  with  the  decision-making  exercise  was 
the  selection  of  "indicators1'  of  performance  and  potential:  although 
many  other  factors  could  have  also  been  included,  the  desire  to  retain 
the  simplicity  of  the  exercise  resulted  in  the  selection  of  three  repre- 
sentative indicators  of  performance  ("job  factors")  and  three  of 
potential  ("personal  factors").  The  rationale  for  the  selection  of  these 
factors  is  presented  in  chapter  three  of  this  report. 

Finally,  the  researcher  acknowledges  the  difficulties  inherent  in 
attempting  to  quantify  officers'  decision-making  policies  regarding 
OER  scores:  in  actual  practice,  an  OER  score  is  assigned  based  on 
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many  factors,  some  of  which  may  not  be  directly  related  to  the  con- 


cepts of  performance  and  potential.  This  research  was  therefore 


designed  to  examine  the  influence  of  performance  and  potential  on 


officers'  judgments  of  OER  ratings,  with  all  other  factors  being  equal. 


Assumptions 


The  assumptions  under  which  this  research  was  conducted  and 


upon  which  conclusions  were  based  are  as  follows: 


1.  Responses  to  the  decision-making  exercise  reflected  honest 
perceptions  of  the  respondents  and  not  "school  answers,  " or 
a gaming  of  the  exercise. 


2.  Job  factors  and  personal  factors  selected  for  use  in  the 
decision-making  exercise  represent  significant  aspects  of 
the  concepts  of  performance  and  potential,  respectively. 


Summary 


In  summary,  this  initial  chapter  has  introduced  the  research 


subject,  identified  the  researcn  problem,  and  has  outlined  the  objec- 


tives, limitations,  and  assumptions  of  the  study.  In  the  remaining 


chapters,  the  methodology  employed  in  investigating  the  research 


questiors  is  presented,  followed  by  a presentation  of  research  results. 


following  se^’^co: 

1.  A literature  search  on  appraisal  systems  and  the  concepts 
of  performance  and  potential  was  conducted. 

2.  A decision-making  instrument  was  developed  to  "capture" 
judgment  policies  of  Air  Force  officers  concerning  perform- 
ance and  potential  criteria  in  appraisal  system  decisions. 

3.  A sample  population  of  Air  Force  officers  was  selected. 

4.  The  decision-making  exercise  was  administered  to  the 
sample  population. 

5.  Collected  data  was  analyzed  to  evaluate  stated  research 
hypotheses. 

6.  Conclusions  and  recommendations  were  made  based  on 
analysis  results. 


Literature  Search 

This  research  was  concerned  with  employee  appraisal  systems-- 
an  area  of  managerial  activity  that  every  Air  Force  officer  has  experi- 
ence in  dealing  with,  either  as  a rater  (of  other  officers)  or  as  a ratee. 
However,  each  officer  may  view  the  officer  evaluation  system  differ- 
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ently  in  terms  of  perceived  evaluation  system  goals,  the  inputs  to  the 
system,  and  the  methods  used  in  administering  the  system. 

Since  this  study  examined  officers'  perceptions  of  the  two 
appraisal  concepts  known  as  performance  and  potential,  a theoretical 
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background  on  the  subject  area  of  appraisal  systems  was  needed.  The 
development  of  a theoretical  background  was  therefore  included  in  this 
research  effort  to  serve  as  (1)  background  information  for  developing 
the  nature  of  the  research  problem,  as  applied  to  both  civilian  and 
military  appraisal  systems,  and  (2)  a theoretical  basis  with  which 
officer  perceptions  of  performance  and  potential  could  be  compared. 

Several  sources  were  used  in  this  assimilation  of  information  on 
appraisal  systems.  The  AFIT  library  provided  a readily  accessible 
source  of  many  periodical  references,  as  well  as  information  on  mili- 
tary applications  of  appraisal  techniques.  The  WrighL  State  University 
library  provided  an  excellent  source  of  personnel  administration  books 
that  were  devoted  to  employee  appraisal.  Additional  information  was 
obtained  from  pertinent  Air  Force  publications  ana  government 
research  studies  obtained  through  the  Defense  Documentation  Center 
(this  particular  source  was  extensively  utilized  to  obtain  information 
on  policy  capturing  techniques). 

Valuable  information  on  Air  Force  research  of  appraisal  systems 
was  obtained  from  the  Air  Force  Human  Resources  Laboratory 
(AFHRL),  Brooks  AFB,  Texas,  particularly  a 1971  AFHRL  report 
titled  USAF  Officer  Evaluation  Systems  Review  and  Research  Recom- 
mendations ■ ^FHRL:1971). 

Apart  from  military  appraisal  systems,  information  on  one 
company's  program  to  identify  executive  potential  was  obtained 
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thr^j^h personal  contact  with  individuals  in  the  Employee  Relations 
Office  of  the  Exxon  Corporation. 

It  is  noted  that  although  the  literature  search  additionally 
involved  a survey  of  available  information  on  policy  capturing  (or 
judgment  modeling)  and  its  application  in  determining  appraisal 
decision  policies,  this  subject  was  considered  to  be  part  of  the 
research  methodology  and  is  therefore  discussed  next  in  this  chapter. 


Policy  Capturing:  Modeling  Human  Judgment 

Since  a rater's  involvement  with  the  officer  evaluation  system 
entails  making  judgments  of  other  officers,  a policy  capturing,  or 
"judgment  modeling"  technique  may  be  employed  to  construct  a mathe 
matical  model  of  the  process  by  which  these  judgments  are  made. 
From  this  model,  one  may  be  a.ble  to  draw  inferences  about  the  influ- 
ence that  various  factors  or  "cues"  have  on  the  decision  process  as 
well  as  the  consistency  with  which  "judges"  make  these  decisions. 

In  a general  sense,  policy  capturing  has  been  defined  by  one 
researcher  as 

. . . the  identification  and  quantification  of  the  attri- 
butes that  are  pertinent  to  a decision  and  the  subsequent 
mathematical  description  of  the  design  policy  for  the 
evaluation  of  these  attributes  (Gooch,  1972). 

An  initial  reaction  of  some  people  to  this  concept  may  be  one  of 
skepticism,  based  on  the  belief  that  human  judgment  processes  are 
too  complex  to  mathematically  model.  However,  an  Air  Force 
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research  study  on  policy  capturing  indicates  just  the  opposite  point  of 
view: 

"...  most  human  decisions  are  not  complex  . . . 
the  variables  in  a decision  may  be  mplex,  but  when 
left  to  the  limited  skills  of  the  human  mind,  the  situation 
will  be  simplified  to  a manageable  surrogate.  [Policy 
Capturing]  systematically  and  efficiently  externalizes 
this  process.  11  (Jones,  £t  al , 1975). 

In  relation  to  employee  appraisal  systems,  the  "policy  of  a 
rater"  has  been  defined  as  "what  raters  do"  when  they  are  asked  to 
respond  to  a series  of  stimuli,  and  "capturing"  a rater's  policy  is 
defining  the  process  of  predicting  the  actions  of  that  rater  from  the 
known  characteristics  of  the  stimuli  he  or  she  is  being  required  to 
evaluate  (Naylor  and  Wherry,  1965:969). 

The  need  for  understanding  how  raters  process  and  evaluate 
information  was  emphasized  by  one  researcher  who  suggested  that 
statistical  techniques  be  used  to  determine  more  clearly  the  basis  of 
policy  decisions,  and  that  the  weights  actually  given  to  criterion 
elements  in  making  judgments  of  performance  should  be  determined 
and  systematized.  Such  techniques,  it  was  arguea,  would  determine 
the  implicit  weighting,  for  a judge  or  rater,  of  the  factors  contributing 
to  assessment  of  others  and  would  permit  a comparison  of  an  individ- 
ual's implicit  policy  with  the  explicit  weighting  policy  adopted  by  the 
organization  (Smith,  1976). 

The  value,  therefore,  of  policy  capturing  is  that  it  permits  the 
inference  of  a rater's  weighting  policy  by  requesting  assessments  of 
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a total  object,  or  an  overall  evaluation  of  a ratee,  rather  than  requir- 
ing explicit  evaluation  of  elements.  The  focus  of  the  analysis  is  on  the 
decision  of  the  rater  rather  than  on  the  rater's  interpretation  of  his  or 
her  decision  processes  (Zedeck  and  Kafry,  1977:287). 


Previous  Research  in  Judgment  Modelin 


There  has  apparently  been  very  little  research  conducted  on  how 
raters  actually  combine,  process,  or  integrate  information  in  forming 
overall  assessments  of  their  ratees.  Three  studies  in  this  subject 
area,  however,  are  known  to  the  researcher. 


Naylor  and  Wherry  (1965)  used  a regression  analysis  for  policy 
capturing  in  a performance  appraisal  situation.  Each  of  50  Air  Force 
supervisors  rated  each  of  250  profiles  (ratees)  described  in  terms  of 
23  traits,  where  each  trait  had  a possible  1-9  range.  The  ~ater  then 
judged  a global  criterion  judgment  on  each  ratee,  and  a regression 
algorithm  was  used  to  describe  the  decision  processes. 

Taylor  and  Wilsled  (1974)  used  a linear  regression  technique  for 
capturing  raters'  policies,  where  the  raters  were  Air  Force  Academy 
cadet  officers.  Results  indicated  that  the  raters  were  internally  con- 
sistent in  their  policies  but  varied  greatly  with  each  other. 

Finally,  a policy  capturing  technique  was  used  in  a study  where 
67  nurses  evaluated  (as  raters)  40  hypothetical  nurses  who  were 
described  by  nine  criterion  elements,  each  having  three  possible 
levels  of  performance  (Zedeck  and  Kafry,  1977).  Overall  evaluations 


were  made  on  a seven  point  scale,  ranging  from  "very  effective"  to 
"very  ineffective."  The  researchers'  analysis  yielded  relative  weights 
for  each  of  the  nine  criterion  elements;  i.  e.  , the  influence  of  each 
element  on  the  overall  evaluation  was  determined,  as  well  as  the  con- 
sistency with  which  raters  made  their  decisions. 

The  usefulness  of  policy  capturing  in  relation  to  employee 
appraisal  is  not  restricted  to  an  evaluation  of  already  existing  systems: 
Kavanagh,  MacKinney,  and  Wolins  (1971)  reviewed  the  content  issue 
in  performance  appraisals;  i.  e.  , the  researchers  proposed  that  prior 
to  implementing  a rating  program  for  performance  appraisal,  judg- 
ments of  the  content  or  stimuli  to  be  rated  must  be  made.  The  content 
is  viewed  as  potentially  ranging  from  subjective  to  objective,  abstract 
to  concrete,  or  personality  to  performance.  Thus,  Kavanagh  proposes 
the  use  of  a policy  capturing  technique  in  the  actual  planning  of  an 
appraisal  system. 

Statistical  Basis  for  a Policy  Capturing  Model 

The  statistical  approach  used  in  this  study  is  based  on  Hoffman's 
proposal  that  it  is  possible  to  model  the  human  judgment  process  by 
representing  the  policy  of  a decision  maker  as  a linear  combination  of 
stimulus  variables,  or  "cues"  (Hoffman,  I960).  Regression  analysis 
is  then  used  to  describe  a judge's  decision  with  the  information  leading 
to  that  decision  by  means  of  correlational  statistics  (Jones,  e£  aJ,  1975). 
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The  basic  approach  requires  the  rater  to  make  quantitative 
evaluations  of  a number  of  ratees,  each  of  whom  is  defined  by  one  or 
more  criterion  elements,  each  of  which  in  turn  has  several  possible 
levels.  If  there  are  a sufficient  number  of  ratees  for  whom  the  rater 
must  provide  an  overall  judgment,  and  each  of  the  ratees  is  described 
by  the  same  set  of  elements  (though  different  levels),  then  a regres- 
sion equation  can  be  developed  for  a rater  which  describes  his  or  her 
idiosyncratic  method  of  combining  and  weighting  information.  The 
mathematical  model  represents  the  strategy  or  policy  of  the  rater;  the 
relationship  between  each  criterion  element  and  the  overall  assess- 
ment, the  beta  weight,  identifies  the  relative  degree  cf  importance  of 
the  criterion  elements  as  they  influence  the  rater's  decisions. 

The  regression  model  takes  the  following  form: 

Y = 0Q  + + ...  + iS.X.  + « 

Y is  the  (dependent)  variable  that  represents  the  decision  being 
modeled;  the  /?s  are  parameters  of  the  regression  model;  the  X^s  are 
the  predictor  variables  (stimuli,  or  ''cues"  in  policy  capturing);  and 

represents  an  error  term  (accounts  for  any  variance  not  explained 
by  the  predictor  vaiiables). 

By  fitting  empirical  data  (various  values  of  the  decision  variable 

Y for  given  levels  of  the  predictor  variables  (X^))  to  a linear  equation 
by  the  least  squares  regression  model,  the  (standardized)  can  be 
determined. 
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Hoffman  (I960)  points  out  that  the  standardized  beta  weights 
signify  the  importance  attached  to  each  of  the  predictor  variables  by 
the  judge.  Large  coefficients  mean,  empirically,  that  the  correspond- 
ing predictors  can  account  lor  a large  proportion  of  the  variance  of 
judgment;  conversely,  a predictor  with  a small  beta  weight  contributes 
little  beyond  the  contribution  of  other  predictors.  Hoffman  points  out, 
however,  that  there  are  limitations  to  the  "characterization"  of  the 
judgment  process  by  means  of  beta  weights:  beta  coefficients  do  not 
account  for  all  of  the  predictable  variance  and  do  not  allow  for  the 


assessment  of  the  relative  contribution  of  each  predictor.  He  then 
argues  that  a set  of  weights  are  desired  which  are  theoretically  capa- 
ble of  accounting  for  all  of  the  predictable  variance  and  which  carry 
exact  interpretations  in  terms  of  components  of  variance  (Hoffman, 
1960:119-120). 

Therefore,  in  order  to  determine  the  relative  influence  of  each 
predictor  variable,  or  cue,  on  the  decision  made  by  a judge  (i.e.,  a 
corresponding  value  of  the  decision  variable  "Y"),  Hoffman  defined  an 
index  termed  "relative  weight.  " It  is  defined  as  follows: 


RW. 


RW\y  is  the  relative  weight  of  predictor  X^;  *iy  is  the  standardized 
regression  coefficient  (beta  weight)  of  predictor  X.  regressed  on  the 
decision  variable  Y;  r.  is  the  intercorrelation  coefficient  between  the 

iy 


20 


....  . 


predictor  variable  X.  and  the  decision  variable  Y;  and  R is  the 

1 y 

squared  multiple  correlation  coefficient  of  the  regression  equation. 
Hoffman  emphasizes  that  "relative  weight"  provides  a means  of  por- 
traying the  relative  contributions  of  each  of  the  predictors  such  that  a 
simple  sum  of  them  accounts  entirely  and  unambiguously  for  the  pre- 
dictable variance  (Hoffman,  1962:78). 

If  the  policy  capturing  technique  involves  the  use  of  orthogonal 
predictors  in  the  regression  the  beta  weights  and  intercorrelation  coef- 
ficients are  identical;  thus,  the  relative  weight  equation  reduces  to  the 


following: 


$■  2 
- ^iy 


ly  R 2 

Ry 


The  orthogt  ality  issue  does  not  imply  that  the  predictors  are  inde- 
pendent (i.  e. , no  interaction),  but  rather  refers  to  the  design  nature 


of  the  instrument.  Jones,  et  al , point  out  that  the  requirement  for 
design  orthogonality  forced  those  involved  with  the  investigation  of 
human  judgment  modeling  to  turn  to  factorially  designed  experiments 
in  order  to  obtain  orthogonal  predictors  (Jones,  _et  al,  1975:19).  This 
research  involved  the  use  of  a policy  capturing  instrument  with  ortho- 
gonal predictors,  the  development  of  which  will  be  discussed  later  in 
this  chapter. 

Thus,  the  policy  capturing  analysis  yields  the  relative  degree 
to  which  each  predictor  variable  influences  the  rater's  decision,  and 


2 

the  calculated  R^  values  are  interpreted  as  measures  of  the  con- 
sistency of  judges  in  applying  decision  policies  (Jones,  _et  al,  1975:18). 
It  is  noted  that  both  individual  and  group  decision  policies  can  be 
mathematically  modeled  using  the  policy  capturing  technique,  although 

the  concept  of  relative  weight  is  meaningful  only  in  the  case  of  individ- 

2 

ual  models.  Also,  "R  " as  a measure  of  consistency  applies  to  both 

2 

individuals  and  groups,  although  one  would  expect  group  R s to  be 
lower  than  individual  R a,  since  raters  may  be  internally  consistent 
in  their  decision  policies  yet  differ  widely  with  other  raters  (Madden, 
1963:2). 

Development  of  the  Decision-making  Exercise 

Part  I of  the  exercise  was  designed  to  collect  standard  demo- 
graphic data  on  respondents  such  as  grade,  aeronautical  rating,  and 
OER  rating  history.  The  demographic  data  base  was  established  in 
order  to  examine  possible  correlations  between  these  factors  and  a 
judge's  decision-making  policies. 

Part  II  of  the  decision-making  exercise  was  designed  to  describe 
a linear  judgment  model  of  the  form 

Y = 3n  + 3 X + X + € 

H0  1 ^2  2 

The  Decision  Variable.  Y is  the  decision,  or  exercise  response 
value,  which  indicates  a respondent's  judgment  (on  a five  point  Likert 
scale)  of  the  appropriateness  of  an  OIK  rating  assigned  to  a 
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hypothetical  officer.  This  method  allows  judgments  to  be  "captured" 
relative  to  overall  perc  .ptions  of  factors  that  may  determine  an  OER 


rating,  regardless  of  whether  the  judge  views  the  situation  from  a 
rater  or  ratee  point  of  view.  An  alternative  approach  was  to  have 
respondents  assign  OER  scores  to  hypothetical  officers;  it  was  deter- 
mined, however,  that  the  total  rating  process  in  the  Air  Force  is  quite 
complex--involving  several  raters  and  perhaps  requiring  several 
iterations  before  a final  rating  is  decided  upon.  It  would  have  there- 
fore been  very  difficult  to  develop  a brief  exercise  which  simulated 
this  process. 

A simpler  "decision"  was  selected  to  investigate  the  relative 
weights  of  the  two  predictors  (performance  and  potential)  in  judgments 
of  OER  scores.  The  decision  was  as  follows:  given  a hypothetical 
officer  with  an  identified  level  of  "performance,  " as  well  as  the  level 
of  some  other  indicators  of  "potential,  " judge  the  appropriateness  of  a 
given  OER  rating  (the  rating  in  the  OER  section  "Evaluation  of 
Potential").  Respondents  were  then  asked  to  select  one  of  five 
responses  on  a five-point  Likert  scale. 

Predictor  Variables.  X^  and  X 2 are  predictor  variables  which 
represent  the  following: 

X^  - Performance  of  a ratee,  as  described  by  three  indi- 
cators ("job  factors")  of  job  performance  during  the 
reporting  period. 

X2  - Potential  of  a ratee,  as  described  by  three  indicators 
("personal  factors")  of  potential  other  than  job  per- 
formance. 
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This  exercise  utilized  three  indicators  of  each  predictor  since 
one  can  not  easily  describe  a level  of  performance  or  potential  with 
one  simple  statement.  Rationale  used  in  selecting  these  indicators  is 
discussed  in  chapter  three. 

Three  levels  were  selected  for  each  predictor  variable  (X^  and 
X^);  i.  e. , all  three  "indicators"  for  each  predictor  were  either  (1) 
good,  (2)  neither  good  nor  bad,  or  (3)  bad. 

The  specific  indicators  used  for  each  predictor  (performance 
and  potential)  were  as  follows: 

X^:  Job  Factors  (indicators  of  performance) 

1.  Accomplishment  of  assignments 

2.  Efficiency  in  the  use  of  resources 

3.  On-the-job  communicative  ability  and  effectiveness 

X^:  Personal  Factors  (indicators  of  potential,  exclusive  of 

job  performance) 

1.  Motivation  and  dedication 

2.  Intellectual  ability 

3.  Sensitivity  to  people  and  their  needs. 

Again,  it  is  emphasized  that  three  levels  were  used  for  each 
predictor  "set,  " and  these  levels  were  assigned  values  of  +1  ("good"), 
0 ("neither  good  nor  bad"),  and  -1  ("bad"). 

The  exercise  was  designed  to  represent  a factorially  designed 
experiment;  i.  e.  , all  possible  combinations  of  predictor  variables 
(performance  and  potential)  are  represented.  Thus,  design 
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orthogonality  was  ensured  and  the  concept  of  relative  weight  is 
applicable. 

Exercise  Format.  Cases  were  constructed  by  combining  each 
level  of  the  X^  variable  with  all  possible  levels  of  the  X^  variable. 
Since  there  were  three  levels  for  each  of  the  two  cues,  there  were 
3X3,  or  9 combinations  possible.  However,  since  the  OER  rating 
assigned  to  each  scenario  was  given  as  either  being  a "1"  ("good")  or 
a "3"  (considered  "bad"  under  the  most  current  OER  system),  there 
were  3X3X2,  or  18,cases  which  were  presented.  The  nine  cases 
corresponding  to  each  OER  score  were  analyzed  separately  to  deter- 
mine if  the  policies  of  judges  varied  when  considering  "good"  or  "bad" 
OER  ratings.  An  example  of  the  decision-making  exercise  scenario 
used  in  this  research  is  presented  in  Figure  1. 

An  additional  assumption  made  in  this  exercise  to  prevent  con- 
fusion in  the  minds  of  the  respondents  was  that  there  was  agreement 
among  the  rater,  additional  rater,  and  reviewer  regarding  the  OER 
rating  given  for  each  scenario.  This  assumption  simplified  the 
decision  to  be  made,  in  terms  of  removing  the  possibility  of  respond- 
ents considering  different  levels  of  OER  review. 

The  18  scenarios  were  ordered  randomly  in  the  exercise  by- 
using  a random  number  table  (Freund,  1971:445),  and  correspond  to 
the  following  levels  for  each  predictor  variable  and  the  OER  rating 
assigned  (Table  2.  1). 
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A.  JOB  FACTORS: 

1.  Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their 
needs. 

C.  An  OER  rating  of  "l"  was  given. 

DECISION  #1 

How  appropriate  is  the  OER  rating  that  was  given?  (Circle 
one  number.  ) 

1 2 3 4 5 

Definitely  Probably  Cannot  Probably  Definitely 

NOT  NOT  Determine  Appropriate  Appropriate 

Appropriate  Appropriate 

Figure  1.  Example  Scenario 

Part  III  of  the  exercise  was  not  part  of  the  policy  capturing 
technique,  but  rather  was  an  attempt  to  gauge  respondents'  reactions 
to  four  commonly  held  beliefs  regarding  what  an  OER  rating  means 
(two  were  performance-related  and  two  were  potential- related).  The 
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Scenario 

Number 


OER 

Rating 


Table  2.  I 

Scenario  Cue  Levels 

Job  Factors 


Personal  Factors 


;rformance) 

(Potential) 

Low 

High 

Low 

Low 

Medium 

Low 

Low 

High 

High 

Low 

High 

Medium 

Medium 

Medium 

High 

High 

Low 

Medium 

Medium 

High 

Low 

Low 

Medium 

Medium 

Low 

Medium 

Medium 

High 

Medium 

Low 

High 

Low 

High 

Low 

High 

Medium 

purpose  of  Part  III  was  to  obtain  additional  data  relating  OER  score, 
performance,  and  potential  in  order  to  make  comparisons  with  the 
policy  capturing  data.  The  Part  III  data  is  more  direct  and  conceptually 
simple  than  the  policy  capturing  information,  and  may  be  considered 
more  of  an  intuitive  reaction  of  officers  to  OER  ratings. 

These  four  "beliefs"  about  OER  scores  were  presented  in  the 
context  of  four  different  situations:  receiving  a "1"  rating,  receiving 
a "3'1  rating,  assigning  {as  a rater)  a "1"  rating,  and  assigning  a "3" 
rating.  The  four  areas  were  related  to  (1)  promotability,  (2)  effective- 
ness at  higher  level  jobs,  (3)  how  well  the  job  was  done,  and  (4)  job 
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performance  relative  to  others.  The  first  two  beliefs  relate  to 
potential,  while  the  second  two  concern  job  performance. 

Respondents  were  then  asked  to  give  their  reactions  to  each  of 
the  four  statements  on  a five  point  scale  indicating  agreement  or 
disagreement.  An  example  of  one  of  the  four  OER  situations  used  in 
this  part  of  the  exercise  is  presented  in  Figure  2.  As  in  Part  II, 
statements  were  ordered  randomly  in  the  actual  exercise. 

The  complete  decision-making  exercise  used  in  this  research  is 
located  in  Appendix  A. 


Selection  of  the  Sample  Population 

The  necessity  for  having  a sample  population  that  was  readily 
accessible  and  convenient  to  survey,  resulted  in  the  following  sample 
population  composition  of  active  duty  Air  Force  officers: 

1.  A random  sample  of  186  Squadron  Officer  School  (SOS) 
students  (Lieutenants  and  Captains). 

2.  A random  sample  of  204  Air  Command  and  Staff  College 
(ACSC)  students  (Captains  and  Majors), 

3.  All  (USAF)  Air  War  College  (AWC)  students  (194) 
(Lieutenant  Colonels  and  Colonels). 

One  might  argue  that  the  sample  population  has  an  inherent  bias 


in  that  the  schools  selected  are  generally  regarded  as  having  students 
who  are  "above  average"  officers.  The  researcher  acknowledges  this 
possibility,  yet  it  is  pointed  out  that  if  inconsistencies  in  judgment 
policies  are  found  for  "above  average"  officers,  then  there  is 
intuitively  an  even  greater  chance  of  average,  or  below  average. 


A.  You  have  just  received  an  OER  rating  of  "3.  " For  each  of  the 
possible  reactions  listed  below,  circle  the  number  which  best 
describes  your  feeling  toward  each  reaction. 

1.  They  (my  boss,  my  chain  of  command,  the  "system")  don't 
think  I should  be  promoted. 


1 

2 

3 4 

5 

Definitely 

Probably 

Unsure  Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

2.  They  are  saying  that  most  other 
better  than  I did  mine. 

people  performed  their 

1 

2 3 

4 

5 

Definitely 

Probably  Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

3.  They  don't  think  I could  be  effective  in  higher 

level  jobs. 

1 

2 

3 

4 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

4.  They 

are  saying  I didn't  do  my  job  very  well. 

1 

2 

3 4 

5 

Definitely 

P robably 

Unsure  Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

Figure  2.  Example  Part  III  Scenario 


officers  exhibiting  judgment  inconsistencies. 


.AFB,  Ohio.  Descriptive  statistics  programs,  group  regressions,  and 
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paired  sample  t-tests  incorporated  in  the  analysis  were  accomplished 
using  the  Statistical  Package  for  the  Social  Sciences,  known  as  SPSS 
(Nie,  jst  ai . , 1975). 

Individual  regression  equations  and  associated  statistics  were 
formulated  using  a Fortran  program  written  by  Captain  Harold  E. 
Klick,  also  a graduate  systems  management  system.  This  program 
calculated,  in  addition  to  individual  regression  equation  coefficients. 
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the  relative  weights  for  each  predictor  variable  and  the  multiple  corre- 

2 

lation  coefficient  (R  ) for  each  exercise  respondent.  This  information 
was  then  stored  on  a disk  file  for  use  in  later  computations. 

Specific.  Analyses  Performed  on  the  Data 

requencies  on  Exercise  Responses.  In  order  to  provide  a 
method  for  examining  the  raw  data  base,  the  SPSS  program 
''frequencies"  was  used  (Nie,  _et  al , 1975:194)  to  provide  frequency 
distributions  on  the  ’-espouses  to  each  exercise  question.  This  output 
provided  a concise  descriptive  analysis  of  the  exercise  respondents 
according  to  grads,  aeronautical  rating,  and  other  demographic 
variables,  as  wall  as  the  distributions  of  the  decision  exercise 
responses. 


Individual  Regression  Analyses.  Two  regression  analyses  were 
performed  (using  the  Fortran  program)  with  the  data  from  each  exer- 
cise respondent:  one  regression  was  accomplished  using  the  nine 
scenarios  wherein  the  assigned  OER  was  a "1,"  and  the  second  regres- 
sion was  accomplished  using  the  remaining  nine  scenarios  in  which  the 
assigned  OER  score  was  a "3."  In  each  regression,  the  degree  of 
appropriateness  of  the  assigned  OER  (the  exercise  response;  was 
regressed  on  the  two  decision  cues:  (1)  performance  and  (2)  other 
indicators  of  potential.  For  each  OER  "set,"  this  program  yielded  the 

relative  weights  that  each  officer  placed  on  performance  and  potential 

2 

factors,  as  well  as  the  R value  associated  with  the  decisions  made. 
Mean  values  of  these  parameters  for  different  groups  of  officers  were 
calculated  and  are  presented  as  research  results. 

Paired  Sample  t-Test  of  Individual  Relative  Weights.  In  this 
research,  there  was  not  an  a priori  assumption  that  officers  use  a 
single  decision-making  policy  when  making  judgments  of  both  "good" 

(a  "1"  rating)  and  "bad"  (a  "3"  rating)  CER  ratings.  Rather,  the  two 
sets  ox  scenarios  were  treated  separately  (as  described  in  the  preced- 
ing section),  and  a paired  sample  t-test  (Nie,  ej:  al,  1975:272)  was 
conducted  on  the  two  sets  of  data  in  order  to  determine  if  mean  values 
of  relative  weights  were  different  for  the  two  given  OER  levels.  This 
analysis  therefore  indicated  whether  officers  used  different  decision- 


making policies  in  judging  "good"  and  "bad"  OER  ratings. 
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Group  B.egression  Analyses.  In  addition  to  individual  decision- 


making policies,  this  research  also  involved  an  examination  of  the 
judgment  policies  of  groups  of  officers.  In  this  regard,  the  degree  of 
appropriateness  of  the  given  OER  score  (the  decision  variable)  was 
regressed  on  the  performance  and  potential  factors,  based  on  the  total 
number  of  decisions  made  by  the  following  groups  of  officers: 

1.  All  groups  (381  respondents) 

2.  SOS  students  (113) 

3.  ACSC  students  (144) 

4.  AWC  students  (124) 

As  in  the  case  of  the  individual  regression  analyses,  separate 
regressions  were  run  on  the  scenarios  with  the  "1"  and  "3"  ratings. 

The  different  combinations  of  groups  were  regressed  in  order  to  pro- 
vide information  to  test  the  hypotheses  that  various  groups  used 
different  judgment  policies  about  OER  scores.  This  analysis,  which 
was  accomplished  using  an  F-test  for  comparison  oi  regression 
models,  is  discussed  in  the  next  section  of  this  chapter. 

The  output  of  the  group  regression  analyses  yielded  standardized 

beta  weights  for  each  predictor  (performance  and  potential),  the  group 
■> 

R“,  and  analysis  of  variance  (ANOYA)  data  used  in  later  F-test  compu- 
tations. 

F-Test  for  Comparison  of  Regression  Models.  The  F-test,  as 
described  by  G.  C.  Chow  (1960:599)  was  used  to  test  whether  there 
was  a statistically  significant  difference  (at  the  .05  level)  in 
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regression  models  being  compared.  The  null  hypothesis  in  each 
comparison  states  that  there  is  no  significant  difference  in  the  regres- 
sion coefficients  c£  the  models  being  compared,  versus  an  alternate 
hypothesis  that  there  is  at  least  one  model  among  those  being  com- 
pared whose  regression  coefficients  are  different.  When  mere  than 
two  models  are  being  compared,  however,  this  test  does  not  identify 
which  model(s)  are  different;  therefore,  when  a null  hypothesis  is 
rejected  in  such  a case,  the  number  of  models  being  compared  in  sub- 
sequent analysis  may  be  partitioned  into  smaller  groups  in  order  to 
more  closely  identify  which  models  differ  from  others.  This  approach 
was  used  in  this  research  effort,  as  indicated  in  the  results  chapter  of 
this  report. 

The  F-test  of  significance  was  used  not  only  to  compare  regres- 
sion models  based  on  the  PME  school  of  respondents,  but  also  involved 
hypothesis  testing  with  groupings  of  respondents  based  on  demographics 
such  as  aeronautical  rating,  recent  OFR  rating,  and  respondents' 


history  as  an  OER  rater. 

Examination  of  Interactive  versus  a Non-interactive  Model. 

-A  . ugh  this  research  was  primarily  involved  with  judgment  modeling 
based  on  an  additive  effect  of  predictor  variables  on  the  decision  valu- 
able, some  researchers  have  proposed  that  higher  order,  or  ''inter- 
action, " terms  should  also  be  included  in  :he  mode's  (Jones,  ed  ai, 
1975:21).  In  essence,  these  higher  order  terms  (e.  g. , an  (Xj  x X£> 
term)  may  be  added  to  the  regression  model  in  order  to  account  for 
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interaction  between  predictor  variables,  therefore  adding  explanative 

power  to  the  model.  Since  there  were  two  predictors  used  in  this 

research,  one  interaction  term  (performance  level  times  the  potential 

level)  was  added  to  each  regression  model  in  order  to  determine  the 

statistical  significance  of  the  interaction  term,  as  well  as  the  resultant 

2 

increase  in  group  R . By  examining  the  relative  contribution  of  inter- 
action terms,  one  may  then  make  observations  about  the  "complete- 
ness" (or  lack  thereof)  of  the  non-interactive  model  in  predicting 
judgments. 

Paired  Sample  t-Test  on  Part  III  Data.  Part  III  of  the  exercise 
was  included  to  obtain  more  direct  reactions  to  OER  scores  with  which 
che  results  of  the  policy  capturing  model  could  be  compared. 

As  described  earlier  in  this  chapter,  there  were  four  situations 
presented  in  Part  III:  awarding  a "1,  " receiving  a "1,  " awarding  a 
"3,"  and  receiving  a "3.  " Following  each  situation,  there  were  four 
commonly  held  beliefs  stated  about  OER  scores  with  which  the 
respondent  was  asked  to  indicate  either  agreement  or  disagreement 
(on  a five  point  scale).  In  each  situation,  two  of  th^  "beliefs"  were 
oerformance  oriented,  and  two  were  potential  oriented. 

In  order  to  determine  which  concept  received  the  most  "agree- 
ment, " a paired-sample  t-test  was  conducted  to  examine  whether  the 
agreement  with  either  potential  beliefs  or  performance  beliefs  was 
greater  than  the  other.  It  was  hypothesized  that  the  relative  weights 
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III  Appraisal  Systems:  An  Examination  of  the  Literature 

As  discussed  previously,  this  research  was  concerned  with  the 
Air  Force  officer  evaluation  system,  and  the  possibility  of  officers 
viewing  the  system  differently  in  terms  of  perceived  evaluation  system 
goals,  as  well  as  inputs  to  the  system  (performance  and/or  potential 
evaluation) . 

Although  this  study  dealt  with  a military  appraisal  system,  it 
involved  many  concepts  that  also  apply  to  civilian  appraisal  systems. 

The  following  discussion  of  appraisal  system  development  and  con- 
cepts is  presented  in  order  to  serve  as  background  information  for 
clarifying  the  nature  of  the  research  problem,  and  to  provide  a theoret- 
ical basis  with  which  officer  perceptions  of  performance  and  potential 
(as  determined  by  analysis  efforts)  can  be  compared. 

The  first  topic  discussed  is  the  historical  development  of 
appraisal  systems  in  general,  as  well  as  the  apparent  trends  in 
appraisal  techniques  that  have  been  documented  in  the  literature.  A 
similar  discussion  of  the  Air  Force  officer  evaluation  system  is  then 
presented.  Following  this  historical  development  of  civilian  and  Air 
Force  appraisal  systems,  the  objectives  of  employee  appraisal  are 
described,  followed  by  a discussion  of  what  appraisal  systems  measure. 
Finally,  a representative  list  of  "indicators"  of  both  performance  and 
potential  is  developed,  as  supported  by  management  literature.  These 
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indicators  were  used  in  the  decision-making  exercise  that  formed 
the  basis  of  this  research  effort. 


Historical  Development  and  Trends 

The  managerial  activity  known  as  employee  appraisal  is  not  a 
recent  product  of  the  "technological  age."  References  to  appraisal  of 
managers  can  be  found  in  antiquity:  emperors  of  the  Wei  dynasty 
(AD  221-265)  had  an  "imperial  rater"  whose  task  it  was  to  evaluate  the 
performance  of  the  official  family.  Centuries  later,  Ignatius  Loyola 
established  a system  for  formal  rating  of  the  members  of  the  Jesuit 
Society  (Whisler  & Harper,  1962:423). 

One  of  the  first  uses  of  formal  appraisal  systems  in  the  United 
States  was  by  government  agencies  in  the  late  19th  century,  a move 
induced  by  criticism  of  waste  and  the  spoils  system  in  government. 

The  real  impetus  to  appraisal  in  business,  however,  came  as  the 
result  of  the  work  of  Frederick  Taylor  and  hi  followers  before  World 
War  I.  These  appraisal  systems  were  related  to  various  efficiency 
factors  developed  from  work  simplification  and  time  and  motion  study 
(Koontz,  1971:17). 

As  concern  with  human  relations  aspects  of  the  work  environment 
increased  in  the  1930s  and  1940s,  behavioral  traits  such  as  "ability  to 
get  along  with  others"  tended  to  become  increasingly  prevalent  in 
appraisal  systems. 


Early  appraisal  systems  v'ere  generally  aimed  at  the  evaluation 
of  hourly  workers  rather  than  managers.  Koontz  points  out  that  mana- 
gerial appraisal  systems  largely  introduced  during  World  War  II  and 
immediately  thereafter  had  their  foundation  in  hourly  labor  perform- 
ance appraisal  systems.  Too  many  of  them.,  he  argues,  were  based 
on  worker  qualities  and  attitudes  rather  than  on  performance  (Koontz, 

1971:19). 

One  source  pointed  out  that  management  literature,  especially 
since  World  War  II,  has  been  increasingly  concerned  with  tin  pr^V^m 
of  performance  appraisal;  in  a 1962  collection  of  50  reprint  uiucles 
on  performance  appraisal,  only  8 were  written  before  1951.  "This 
suggests,  " the  authors  pointed  out,  "that  although  performance 
appraisal  is  not  new,  it  has  only  recently  gained  the  attention  it 
deserves"  (Whisler  & Harper,  1962:2). 

With  rising  interest  in  behavioral  science  following  World  War  II, 
there  was  a tendency  toward  a traditional  "trait  approach"  to  employee 
appraisal.  One  study,  conducted  in  1957,  concluded  the  following: 

"Most  companies  . . . are  concerned  mainly  with 
personality  and  cnaracter  traits  ...  so  strongly  is  the 
emphasis  on  personality  that  job  knowledge  and  even  job 
performance  may  have  only  a minor  place  in  the  overall 
rating"  (Dale  and  Smith,  1957:22). 

There  have  been  many  explanations  offered  for  this  "drift" 
toward  such  a trait  approach:  one  analyst  offered  three  underlying 
reasons:  (1)  as  behavioral  sciences  captured  public  interest,  there 
was  a greater  tendency  to  explain  effectiveness  by  psychological  or 
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psychiatric  measures  (2)  since  so  many  managerial  tasks  cannot  be 
given  quantitative  measurement  and  since  qualitative  factors  are  diffi- 
cult to  measure,  there  has  been  a tendency  to  drift  toward  personality- 
centered  appraisals  (3)  the  third  factor  relates  to  the  manager's  job; 
practically  every  study  has  found  successful  managers  to  be  strong 
leaders  and  have  highlighted  human  relations  skills  (Kellogg,  1965:60). 

Beginning  in  the  late  1950s  and  early  1960s,  the  merits  of 
employee  appraisal  were  widely  argued.  Douglas  McGregor,  in  his 
widely  read  article  "An  Uneasy  Look  at  Performance  Appraisal,  " con- 
demned the  traditional  approach  to  appraisal  wherein  managers  were 
put  in  the  "untenable  position  of  judging  the  personal  worth  of  subordi- 
nates, and  then  acting  on  these  judgments.  " (McGregor,  1957:94) 
Other  criticisms  of  traditional  appraisal  systems  were  offered  by 
Likert  (1959),  Stoiz  (I960),  Mayfield  (I960),  and  others. 

McGregor's  alternative  approach  to  appraisal,  which  places  the 
major  responsibility  on  the  subordinate  for  establishing  performance 
goals  and  self  appraising  progress  toward  them,  appears  to  have  its 
basis  in  Drucker's  concept  of  management  by  objectives,  or  "MBO.  " 
Under  such  an  approach,  an  employee  and  hit  or  her  supervisor 
periodically  negotiate  what  the  subordinate  should  accomplish  by  the 
end  of  the  rating  period  (Drucker,  1954:121). 

The  inclusion  of  MBO  concepts  in  appraisal  systems  is  becoming 
increasingly  prevalent:  a recent  study  indicated  that  over  half  of  293 
firms  surveyed  used  some  form  of  MBO  in  their  appraisal  system. 
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The  study  concluded  that  MBO-based  appraisal  methods,  since  its 
measurements  are  clearly  job-related  and  appraisals  are  easier  to 
make:  "...  an  individual  either  meets  his  objectives,  or  he 
doesn't."  (Anderson,  1978:1). 

The  move  to  objective  or  results -oriented  appraisal  has  also 
been  precipitated  by  equal  opportunity  laws:  recent  federal  court  rul- 
ings essentially  have  made  it  the  responsibility  of  an  employer  to  prove 
that  his  rating  system  does  not  discriminate;  companies  must  now 
demonstrate  both  that  their  systems  are  reliable  and  that  the  factors 
that  are  measured  are  truly  related  to  job  performance.  Actual  court 
cases  have  upheld  employee  complaints  of  "patently  subjective"  judg- 


ments based  on  factors  that  had  little  to  do  with  how  che  job  was  done. 
Anderson  thus  concludes  that  as  lower-level  executives  become  increas- 


ingly anonymous  in  the  expanding  white-collar  work  force,  many 
companies  do  not  want  to  rely  on  informal  rating  systems:  they  believe 
they  will  be  in  a better  position  to  avoid  conflict  with  equal  opportunity 
laws  if  they  justify  personnel  decisions  with  uniform  appraisal  stand- 
ards (Anderson,  1978:1). 

Concomitant  with  the  increasing  reliance  on  formal  appraisal, 
however,  is  increasing  criticism  of  most  systems.  There  appears  to 
be  instability  regarding  many  organizations'  appraisal  systems:  a 
recent  report  by  the  Conference  Board  noted  that  over  half  of  the  293 
firms  it  surveyed  had  developed  new  systems  within  the  past  three 
years,  yet  current  systems  were  still  -widely  regarded  as  a nuisance 
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at  best  and  a dangerous  evil  at  worst.  One  consultant  admitted  that 
finding  a workable  appraisal  system  "seems  like  a search  for  the  Holy 
Grail."  (Anderson,  1978:1) 

Evolution  of  the  Air  Force  Officer  Evaluation  System 

The  history  of  the  Air  Force  officer  evaluation  system  is  rather 
short,  but  nevertheless  illustrates  the  evolutionary  process  that  many 
managerial  appraisal  systems  undergo. 

When  it  became  a separate  service  in  1947,  the  Air  Force  con- 
tinued to  use  the  Army  officer  evaluation  system,  which  had  been 
substantially  revised  earlier  in  1947.  Whereas  ..he  Army  system  had 
previously  been  based  on  a trait  approach  to  appraisal,  the  "new"  AGO 
Form  67  required  a different  method  of  evaluation:  the  essence  of  this 
form  was  one  section  which  provided  a rater  with  multiple -choice  ques- 
tions about  ratee  characteristics,  and  an  additional  section  which  pro- 
vided space  for  written  comment®1  by  the  rater  (Rhoades,  et  al,  1978: 
10). 

General  dislike,  however,  of  this  system  prompted  .he  Air  Force 
to  implement  its  own  OER  system  with  the  introduction  of  Air  Force 
Form  77,  15  March  1949.  This  system,  although  based  on  extensive 
research  of  officer  perceptions  of  rating  factors  by  the  American 
Institute  of  Research,  was  also  found  to  be  lacking  in  that  it  was  too 
complex  (there  were  54  items  to  be  scored),  time  consuming,  and  gave 
rise  to  "inflated"  ratings  (AFR  36-10,  1975:A~2). 
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In  1952,  a new  system  was  introduced  that  "survived,  " with 
minor  modifications,  for  22  years.  Two  somewhat  similar  forms 
were  used  with  this  system:  the  Air  Force  Form  77  (used  for  rating 
company  grade  officers)  was  task  oriented,  while  the  >:ir  Force  Form 
707  (used  for  rating  field  grade  officers)  was  management  oriented 
(Rhoades,  et  al,  1976:12). 

The  front  side  of  both  forms  was  used  for  identification  data 
(Section  I),  and  a description  of  duties  (Section  II).  Section  III 
described  eight  ’’rating  factors, ’’  which  were  listed  in  terms  of  a five 
block  scale  ranging  from  "outstanding"  to  "poor"  descriptors.  Section 
IV  entailed  a similar  five  block  rating  of  "military  qualities."  Section 
V,  which  was  titled  "Overall  Evaluation,  " involved  a ten  block  scale 
ranging  from  "unsatisfactory"  to  "outstanding.  " 

Section  VI  involved  a rating  of  "Promotion  Potential,  " on  a four 
block  "descriptor"  scale.  Finally,  Section  VII  provided  space  for 
rater  comments,  and  an  additional  section  provided  space  for  com- 
ments by  an  indorsing  official.  The  Air  Force  Form  77,  November 
1966,  can  be  found  in  Appendix  C. 

Although  this  system  lasted  for  22  years,  problems  with  rating 
"inflation"  led  to  its  demise.  In  1961,  less  than  five  percent  of  all  Air 
Force  Officers  were  receiving  "top  block"  ratings  in  the  "Promotion 
Potential"  section:  in  1974,  over  90  percent  had  received  top  ratings 
five  consecutive  times  (Rhoades,  £t  al,  1976:1). 
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In  November  1970,  the  Director  of  Personnel  at  Headquarters 
USAF  requested  that  the  Air  Force  Human  Resources  Laboratory 
(AFHRL)  develop  a new  officer  evaluation  system,  based  on  the  belief 
that  the  current  system  was  inadequate  for  differentiating  between 
individuals  for  selection  (promotion)  and  assignment  purposes 
(Johnson,  et  al,  1976:5). 

The  proposed  OER  syo.em  was  designed  by  the  AFHRL  around  a 
management  by  objectives  concept:  the  system  was  based  on  a state- 
ment of  job  objectives  and  was  designed  to  assure  some  degree  of 
counseling  in  the  rating  process.  Additionally,  a separate  evaluation 
of  promotion  potential  was  recommended,  since  AFHRL  researchers 
questioned  the  capability  of  a "single  system  doing  everything  for 
everybody."  (Johnson,  e_tal,  1976:5,  8).  The  AFHRL  recommenda- 
tions specifically  stated  that  the  Air  Force  GES  objectives  should  be  to 
produce  valid  and  discriminating  assessments  of  (1)  performance  and 
(2)  potential,  in  order  to  provide  information  for  assignment,  counsel- 
ing, and  selection  (/FHRL,  1971:App6).  Additional  emphasis  on 
potential  as  well  as  performance  assessment  was  stressed  in  the 
"personnel  plan,"  approved  by  the  Secretary  of  the  Air  Force,  which 
stated  that  promotion  is  to  be  based  on  performance  and  potential- -in 
the  past,  emphasis  has  been  almost  entirely  on  past  performance 
(AFHRL,  1971:Tab2,  App  2). 

The  OER  review  group,  which  considered  the  recommendations 
of  the  AFHRL,  substantially  modified  the  proposed  system:  the  MBO 
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and  the  "two  forrr"  concepts  were  eliminated,  as  was  the  requirement 
for  interaction  between  the  rater  and  ratee  regarding  the  appraisal  of 
job  accomplishments  (Johnson,  et  al,  1976:10,  11). 

More  specifically,  the  current  OER  system  does  not  outline  this 
"developmental"  function  as  an  objective  of  the  system:  AFR  36-10 


states  the  following: 

"Evaluation  reports  are  designed  for  the  personnel 
management  of  Air  Force  Officers.  They  are  not  to  be 
used  as  counseling  devices  . . . counseling  is  per- 
formed (continuously)  by  the  supervisor  for  the  purpose 
of  assisting  individuals  in  improving  their  performance." 
(AFR  36-10,  19  78:1-2) 


The  revised  OER  system  was  implemented  by  the  Air  Force  in 
November  1974.  The  new  Air  Foxce  Form  707,  November  1974  has 


two  evaluation  sections:  Part  III  involves  a rating  of  performance 
factors  similar  to  that  found  in  the  previous  system  except  that  it  has 
no  overall  evaluation  of  performance,  and  Part  IV  is  titled  "Evaluation 
of  Potential."  lhis  evaluation  of  potential  is  scored  in  terms  of  a six 
block  scale  ranging  from  "highest"  to  "lowest."  In  order  to  control 
rating  inflation  in  this  section,  a controlled  distribution  was  imposed 
on  the  top  two  blocks  of  the  rating  of  potential  (AFR  36-10,  1975:5-3). 
For  clarification  purposes,  a "top  block"  rating  in  Section  IV  is  com- 
monly referred  to  as  a "1,  " the  second  level  block  a "2,  " etc.  (through 
"6,"  the  lowest  rating  possible). 


In  January,  1978,  !he  controlled  rating  distribution  was  modified 
to  reflect  limitations  only  on  the  percentage  of  top  block  ratings 


awarded  (AFR  36-10,  1978:6-1).  Since  the  current  controlled  rating 
distribution  only  applies  to  the  "evaluation  of  potential,"  some  writers 
have  keyed  on  the  assessment  of  potential  as  the  essence  or  heart  of 
the  current  OER,  such  as  Blakelock  ( 1976:30),  Rhoades,  et  al  (1972:6). 
and  Dunne  (1977:12).  The  concept  of  "potential"  assessment  versus 
"performance"  assessment  leads  to  an  examination  of  some  of  the 
problems  associated  with  appraisal  systems.  The  remainder  of  this 
discussion  will  therefore  focus  on  the  sources  of  these  problems:  the 
objectives  of  appraisal  systems,  and  the  question  of  what  the  systems 
are  designed  to  appraise. 

The  Objectives  of  Employee  Appraisal 

Peter  Drucker,  in  his  discussion  of  the  need  for  employee 
appraisal,  pointed  out  that  "insistence  on  high  goals  and  high  perform- 
ance requires  that  a man's  ability  both  to  set  goals  and  to  attain  them 
be  systematically  appraised."  He  additionally  pointed  out  that  since 
managers  make  many  decisions  ba  ^d  on  appraisals  of  employees, 
these  managers  need  a systematic  appraisal  or  else  too  much  time  is 
wasted  on  making  decisions  by  "hunch"  rather  than  by  knowledge 
(Drucker,  1954:149). 

Kellogg  states  that  to  appraise  anything  is  to  set  a value  on  it; 
however,  there  is  no  such  thing  as  a universally  accepted  value.  She 
additionally  proposes  that  managers  should  discard  the  notion  that 
appraisals  are  "absolute"  evaluations  with  which  associates  will  agree: 
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he  or  she  should  recognize,  instead,  that  appraisal  is  a subjective 


judgment  made  on  the  basis  of  information  which  is  incomplete 
(Kellogg,  1965:15).  This  inherent  subjectivity  is  reflected  in  the  fol- 
lowing definition  of  employee  appraisal: 

"Employee  appraisal  is  a manager's  subjective  judgment 
of  the  value  of  an  individual's  ability  to  do  something:  the 
managei  judges  one's  present  and/or  future,  usefulness  to  a 
job  or  a business.  " (Kellogg,  1965:19) 

Additionally,  Kellogg  points  out  that  employee  appraisal  is  most 
likely  to  be  sound  if  its  purpose  is  well  defined  (Kellogg,  1965:19).  A 
discussion  of  the  objectives  of  employee  appraisal,  therefore,  is  pre- 
sented at  this  point. 

In  any  managerial  activity,  one  must  initially  consider  the  goals 
of  that  activity.  In  this  regard,  an  examination  of  the  goals  of 
employee  appraisal  is  in  order.  This  discussion  is  considered,  essen- 
tial, as  one  source  indicated  that  a fundamental  reason  for  the  failure 
of  performance  appraisal  systems  is  the  failure  of  management  to 
clearly  define  objectives  and  to  establish  techniques  specifically 
designed  to  accomplish  these  objectives  (Huse,  1967:3). 

In  general,  the  primary  purpose  of  employee  appraisal  is  to 
facilitate  improved  results.  In  an  attempt  to  accomplish  organizational 
goals,  managers  are  actively  involved  in  direct  contact  with  subordi- 
nate managers  and  workers  by  executing  the  "directing"  and  "control- 
ling" managerial  functions.  In  this  regard,  employee  appraisal  results 

in  an  "information  base"  on  employees  that  is  used  for  two  distinct 
purposes. 
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The  first  of  these  is  personnel  administration,  wherein  the 
manager  evaluates  subordinates  regarding  their  performance  (and 
possibly  potential  for  future  performance),  and  subsequently  forms 
the  basis  for  decisions  concerning  promotions,  transfers,  and  merit 
rewards,  including  salary  administration  (Cummings,  1973:4). 

In  order  to  "improve  results"  in  the  short  run,  managers  evalu- 
ate the  performance  of  subordinates  and  take  necessary  actions  to 
promote  better  performance.  In  the  long  run,  however,  managers 
generally  consider  (and  evaluate,  at  least  mentally)  both  the  potential 
of  an  employee  (future  worth  to  the  organization)  and  the  development 
of  employees  to  fulfill  future  needs  of  the  organization.  The  concepts 
of  performance  and  potential  will  be  examined  in  more  detail  in  a 
subsequent  section  of  this  chapter,  so  discussion  will  now  focus  on  the 
second  use  of  appraisal  information:  personal  development. 

From  this  viewpoint,  appraisals  are  designed  to  help  improve 
performance  or  the  capability  for  performance  directly  by  aiding  the 
employee  in  identifying  areas  for  improvement  and  growth  (Cummings, 
1973:4-5).  This  developmental  function  is  essential  to  both  the 
subordinate  (for  personal  growth  and  increased  job  competence)  and  the 
organization.  If,  through  the  appraisal  process,  managers  are  success 
ful  in  further  developing  an  employee's  effectiveness,  organizational 
goal  accomplishment  will  have  been  facilitated. 

This  developmental  approach  therefore  emphasizes  the  import- 
ance of  managerial  actions  in  improving  and  facilitating  an  individual's 
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effectiveness  relative  to  his  or  her  own  abilities  (Cummings,  1973:6). 

The  manager  is  therefore  placed  in  a counseling  role  in  his  or  her 
relationship  with  subordinates.  Terry  emphasizes  that  it  is  the  man- 
ager's job  to  guide  subordinates,  not  to  inspect  and  drive  them  toward 
sought  for  improvements  (Terry,  1974:226). 

Employee  appraisal  therefore  establishes  an  information  base 
upon  which  personnel  administration  and  personal  development  actions 
are  hopefully  taken  to  facilitate  organizational  goal  accomplishment. 

Formal  and  Informal  Appraisal 

Although  many  managers  may  think  of  employee  appraisal  in 
terms  of  formal  systems  (i.  e. , documented  appraisals  on  a specified 
form  at  required  intervals),  there  also  exists  a continuous,  informal 
type  of  appraisal.  The  manager,  through  day-to-day  contacts  with 
subordinates,  continuously  appraises  their  performance  and  takes 
actions  which  hopefully  improve  performance.  Informal  appraisal  can 
be  thought  of  as  one  input  to  the  formal  appraisal  of  suoordinates.  Thus, 
the  informal  appraisal  of  subordinates  involves  maintaining  effective 
communications  lines  between  superiors  and  subordinates  that  serve  as 
routes  for  positive  feedback  as  well  as  fo^  corrective  measures. 

The  two  perspectives  for  viewing  appraisal -personnel  admin- 
istration and  personal  development- -are  applicable  to  both  formal  and 
informal  means  of  appraisal.  Whereas  the  personnel  administration 
viewpoint  is  most  often  associated  with  formal  appraisal  systems,  the 
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personal  development  viewpoint  is  most  frequently  associated  with  the 
manager's  day-to-day,  informal  contact  with  subordinates.  In  some 
appraisal  systems,  however,  personal  development  is  additionally 
accomplished  (at  least  by  design)  through  a formal  appraisal  interview: 
these  appraisal  interviews  are  conducted  periodically  and  involve  a 
discussion  between  the  supervisor  and  the  subordinate  of  the  formal 
performance  rating,  strengths  and  weaknesses,  and  areas  needing 
improvement.  Thus,  the  appraisal  interview  serves  as  an  important 
source  of  feedback  to  the  employee  that  lets  him  or  her  know  where  he 
or  she  stands. 

Although  there  may  be  much  to  be  gained  from  an  informal  sys- 
tem of  employee  appraisal,  it  appears  that  many  managers  are  either 
reluctant  or  ineffective  in  this  regard.  Thus,  major  emphasis  is 
placed  by  many  organizations  on  the  formal  system  of  appraisal,  par- 
ticularly with  respect  to  personnel  administration  and  associated 
decisions  regarding  salary,  promotions,  etc.  This  research  was 
oriented  toward  the  implications  of  formal  appraisal  systems,  particu- 
larly as  related  to  the  Air  Force  officer  evaluation  system. 

A significant  question  to  be  asked  when  considering  appraisal 
systems  concerns  what  the  system  is  intended  to  measure.  This  sub- 
ject is  certainly  essential  to  any  effective  appraisal  system,  for  mana- 
gers must  be  aware  of  and  understand  what  is  being  evaluated  before 
the  appraisal  system  can  be  effective  in  achieving  desired  results. 


49 


•*  xi . 


measure,  as  well  as  some  of  the  obstacles  inherent  in  attempt  to 
subjectively  evaluate  human  performance  (or  the  capability  for  per- 
formance). Particularly  when  dealing  with  formal  appraisal  systems, 
managers  must  be  cognizant  of  what  is  being  measured;  if  managers 
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actually  evaluate  employees  based  on  factors  different  than  those 
intended  by  the  appraisal  system  design,  dysfunctional  organizational 
effects  are  possible  which  limit  the  accomplishment  of  organizational 
goals. 


Performance.  In  many  appraisal  systems,  supervisors  are 
simply  evaluating  employees'  past  performance  (i.e.,  performance 
appraisal).  Since  the  concept  of  performance  is  being  discussed,  a 
reiteration  of  the  definition  presented  in  chapter  one  is  in  order: 

Performance  is  defined  as  the  degree  to  which  an 
employee  has  satisfactorily  accomplished  his  or  her  job 
or  assigned  duties  during  a specified  period  of  time. 

Performance  is  determined  primarily  by  one's  abilities,  coupled 
with  motivational  forces:  i.  e.  , one  must  have  both  the  capability  and  the 
desire  to  do  it.  However,  Cummings  points  out  that  since  performance 
is  an  individual  phenomenon,  environmental  variables  influence  per- 
formance, primarily  through  their  effect  on  the  individual  determinants 


of  performance- -ability  and  motivation,  as  shown  in  the  model  below. 
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Figure  3.  Performance  Model  (Cummings,  1973:9) 


Ability  reflects  capability;  a relatively  stable  characteristic  which 


enables  persons  to  behave  in  some  specified  fashion.  Motivation,  on 


the  other  hand,  reflects  effort  or  energy,  a dynamic  characteristic 


which  determines  how  vigorously  capabilities  will  be  employed  in  some 


activity  (Cummings,  1973:8). 


One  might  argue  that  examining  the  effects  of  environmental 


factors  on  the  determinants  of  performance  is  not  relevant  to  the  sub- 


ject of  appraisal,  since  performance  is  viewed  in  retrospect  (past 


actions  are  considered).  However,  if  some  of  these  environmental 


factors  have  adverse  effects  on  the  determinants  of  performance,  then 


one's  demonstrated  level  of  performance  may  not  be  truly  indicative  of 


an  employee's  true  capabilities  to  perform,  i.  e. , that  employee's 


potential.  Patz  points  out  that  past  performance  is  only  one  indicator 


of  what  future  performance  may  be  at  higher  levels.  Additionally,  he 
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emphasizes  that  promotion  decisions,  like  all  decisions  that  affect  the 
future,  should  reflect  estimates  of  management  potential  more 
strongly  than  appraisals  of  past  performance  (Patz,  1975:79).  Laurent 
emphasizes  that  performance  in  one  job  is,  to  a degree,  predictive  of 
performance  in  another,  but  generally  only  to  the  extent  to  which  the 
two  jobs  are  similar;  the  accuracy  of  the  prediction  decreases  as  the 
jobs  become  more  dissimilar  (Laurent,  1968:3). 

Potential.  As  was  alluded  to  in  earlier  discussion,  managers 
are  not  only  concerned  with  past  performance,  but  are  additionally 
oriented  toward  the  effective  performance  of  subordinates  in  the 
future,  particularly  when  these  subordinates  are  also  managers. 

Some  appraisal  systems  are  therefore  designed  to  predict  the  future 
worth--the  potential--of  individuals  to  the  organization. 

One  private  industry  research  report  emphasized  a primary 
reason  for  the  . . :st  in  identifying  executive,  or  management, 
potential: 

"The  job  of  a manager  in  a business  organization  is 
becoming  more  and  more  complex,  which  means  that 
those  going  into  management  positions  in  the  future  must 
have  more  ability  than  some  of  those  who  were  able  to  do 
an  acceptable  job  in  the  past.  These  needs  put  a premium 
on  the  early  identification  of  employees  who  have  the 
potential  to  be  successful  in  management  positions." 

(Laurent,  1968:1) 

One  survey  of  top  and  middle  managers  in  19  companies  indi- 
cated that  the  purpose  of  appraisal  is  not  only  a developmental  tech- 
nique to  improve  results,  but  is  also  a necessary  vehicle  for 
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assessing  management  potential  (Patz,  1975:75). 

Not  all  management  writers  support  fhe  appraisal  of  individual 
potential,  as  evidenced  by  Drucker's  statement  that  an  appraisal 
which  "focuses  on  potential,  on  personality,  on  promise,  on  anything 
that  is  not  provable  performance,  is  an  abuse."  (Drucker,  1954:150) 
Although  in  1954  openly  opposing  the  rating  of  potential  in  employee 
appraisal  systems,  Drucker  appears  to  lend  some  support  (possibly 
unintentional)  to  the  appraisal  of  potential  in  his  more  recent  book. 

The  Effective  Executive: 

"Effective  executives  know  that  they  have  to  start 
with  what  a man  can  do  rather  than  what  a job  requires. 

This  means  that  they  do  their  thinking  about  people  long 
before  the  decision  on  filling  a job  has  to  be  made,  and 
independently  of  it.  The  purpose  is  to  arrive  at  an 
appraisal  of  a man  before  one  has  to  decide  whether  he 
is  the  right  person  to  fill  a bigger  position."  (Drucker, 

1967:83) 

Drucker's  comment  appears  to  indicate  that  one  cannot  always  wait 
on  proven  performance  to  base  certain  decisions;  one  therefore  makes 
predictions  of  one's  future  performance  based  on  various  factors. 

As  was  mentioned  in  earlier  discussion,  one  of  the  objectives  of 
employee  appraisal  is  to  provide  information  upon  which  promotion 
decisions  are  based.  One  study  indicated  that  some  companies  placed 
major  weight  in  promotion  decisions  on  the  immediate  superior's 
estimate  of  the  subordinate's  potential,  rather  than  on  the  superior's 
estimate  of  the  subordinate's  job  performance  effectiveness  (Campbell, 
et  al,  1970:37). 
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Since  potential  is  future  oriented,  a manager  may  encounter 
difficulties  when  attempting  to  measure  this  individual  capability. 
These  difficulties  are  illustrated  by  the  following  excerpt  from  an 
Exxon  USA  research  report  on  measuring  management  potential: 

"If  the  evaluation  of  performance  is  difficult,  the 
evaluation  of  potential  is  even  more  so.  This  is  particu- 
larly true  when  one  attempts  to  evaluate  the  management 
potential  of  young  employees.  " (Laurent,  1968:2) 

Kellogg  indicate  j that  the  first  and  most  important  thing  for  a manager 
to  do  is  recognize  his  or  her  limitations  in  predicting  the  future  and 
do  everything  possible  to  gather  accurate,  relevant,  representative 
information  as  a basis  for  judgments.  Therefore,  the  manager  makes 
a judgmental  decision  about  the  probability  that  a given  employee  will 
grow  at  a rate  which  will  place  him  or  her  in  competition  for  certain 
key  openings  in  the  future.  Kellogg  also  points  out  that  because  of  the 
relative  inadequacy  of  this  decision,  it  should  be  considered  a short 
term  one,  subject  to  frequent  reviews  as  the  future  unfolds  both  for 
the  firm  and  for  the  individual  (Kellogg,  1965:137).  Drucker  also  sup- 
ports this  "short  term"  concept  by  indicating  that  one  cannot  appraise 
potential  for  any  length  of  time  ahead  or  for  anything  very  different 
from  what  a man  is  already  doing  (Drucker,  1967:85). 

Regarding  Air  Force  officer  evaluations,  the  emphasis  also 
appears  to  be  on  a "short  term"  evaluation  of  potential.  AFR  36-10, 
in  discussing  potential,  emphasizes  this  short  range  time  frame: 
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"Evaluators  will  focus  primarily  on  near  term  capa- 
bilities for  increased  responsibilities;  from  an  evaluator's 
perspective,  long  range  assessment  in  a subordinate 
element  for  which  primary  responsibility  rests  with  the 
selection  board."  (AFR  36-10,  1978:2-22) 

Thus,  employee  appraisal  systems  may  be  designed  to  measure 
(past  oriented)  performance  and/or  (future  oriented)  potential  in  sup- 
port of  appraisal  systems  objectives.  The  next  section  of  this  chapter 
presents  several  "indicators"  of  performance  and  potential,  as  sup- 
ported by  various  sources  in  the  management  literature. 


Indicators  of  Performance  and  Potential 

As  was  mentioned  in  earlier  discussion,  measuring  the  perform- 
ance of  managers  is  not  always  a simple  task,  particularly  since  a 
large  proportion  of  managerial  and  professional  jobs  simply  do  not 
have  readily  identifiable  units  of  output.  Thus,  it  appears  that  the 
higher  the  job  level  in  the  organizational  hierarchy,  the  more  abstract 
and  difficult  to  measure  job  performance  becomes.  Furthermore,  the 
evaluation  of  management  potential  appears  to  pose  even  greater  diffi- 
culties since  potential  is  based  on  predictions  of  future  performance, 
rather  than  exclusively  on  assessments  of  past  performance. 

Since  this  research  involved  a policy  capturing  method  of  deter- 
mining officer  perceptions  of  performance  and  potential,  "indicators" 
of  each  conept  were  chosen  for  inclusion  in  the  decision-making  exer- 
cise. In  genera!,  performance  represents  one's  past  accomplishments 
in  terms  of  effectiveness  and  efficiency,  while  potential  is  a more 
abstract  concept  that  is  additionally  based  on  various  personality  traits 
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and  individual  capabilities.  The  following  three  indicators  of  per- 


formance are  considered  a representative,  but  certainly  not  compre- 


hensive, listing  based  on  current  management  literature. 


Performance  Indicators 


1.  Accomplishment  of  assignments --This  indicator,  selected 


to  represent  the  degree  of  on-the-job  accomplishments  during  a rating 


period,  is  perhaps  the  most  obvious  performance  indicator  and  is  sup- 


ported, among  others,  by  Pigors  and  Myers  (1977:47,274). 


2.  Efficiency  in  the  use  of  resources--Not  only  is  performance 


determined  by  how  effective  a job  is  accomplished,  but  also  by  the 


degree  of  efficiency  wir.h  which  resources  are  utilized.  This  reason- 


ing is  supported  by  Mann  and  Dent  (1954:105),  Johnson,  Meehan  and 


Y/ilkinson  (1976:7),  and  efficiency  is  included  as  a performance  factor 


on  the  Air  Force  Officer  Effectiveness  Report  (Air  Force  Form  707). 


3.  Communicative  ability  and  effectiveness --The  importance  of 


being  a proficient  communicator,  both  orally  and  in  written  form,  has 


been  emphasized  by  Kellogg  (1965:142),  Johnson,  £t  al  ( 1976:7),  and  is 


also  included  as  a performance  factor  on  the  OER  form. 


Three  representative  indicators  of  potential  obtained  from  a 


search  of  management  literature  (indicators  other  than  immediate  past 


performance)  are  as  follows: 


if 


m 
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Potential  Indicators 


1.  Individual  motivation  and  dedication- -This  "drive"  factor  is 
listed  in  AFP  36-26,  paragraph  3-2c,e  as  "essential  and  without 
which  an  officer's  potential  is  strictly  limited."  (AFP  36-26,  1977) 
Additional  support  for  this  factor  is  offered  by  Guyton  >1969:659),  and 
Laurent  (1968:3). 

2.  Intellectual  ability- -Guyton  concludes  that  this  factor  is  a 
strong  contributor  to  success  prediction  (Guyton,  1969:660).  Other 
sources  which  support  this  factor  are  Cummings  (1973:16)  and  Laurent 
(1968:3). 

3.  Sensitivity  to  people  and  their  needs--AFP  36-26,  paragraph 
3-2J  specifically  points  out  that  human  relations  is  an  important  aspect 
of  potential.  Koontz  (1971:22)  lends  additional  support  to  the  inclusion 
of  human  relations  ability  as  an  indicator  of  management  potential. 

Although  recent  performance  and  trends  in  past  performance  are 
considered  to  be  valid  predictors  of  potential,  these  factors  were  not 
included  in  the  decision-making  exercise  as  indicators  of  potential  in 
order  to  clearly  distinguish  between  "performance- related 11  and  "other" 
indicators  of  potential.  This  rationale  was  considered  essential  to  the 
research  effort,  since  it  has  been  argued  that  in  some  places  the 
frame  of  reference  in  military  appraisal  systems  is  "potential,  " but 
the  assumption  is  that  potential  is  little  more  than  performance  (Dunne, 
1977:12).  In  order  to  examine  this  argument,  a policy  capturing  model 
was  developed  to  distinguish  between  the  relative  importance  that 


officers  place  on  (1)  past  performance  and  (2)  other  indicators  of 


potential  when  making  judgments  of  OER.  ratings  (Section  IV,  "Evalu- 
ation of  Potential").  The  resulting  weights  that  officers  placed  on 
performance  and  potential  were  used  to  examine  if  potential  is,  in 
fact,  assumed  in  some  situations  to  be  "little  more  than  perform- 
ance," as  Dunne  hypothesizes. 
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IV  Results 

Exercise  Response 

As  indicated  in  Table  4.  1,  a total  of  584  decision-making  exer- 
cises were  distributed  to  students  at  the  three  PMF  schools  at 
Maxwell  AFB,  Alabama.  Of  the  exercises  returned,  381  were  usable, 
resulting  in  an  overall  return  rate  of  65.  2%. 


Table  4.  1 

Exercise  Response  Rates 


PME 

School 

Number  of 
Exercises 
Distributed 

Number  of 
Usable 
Returns 

Return 

Rate 

SOS 

186 

113 

60.  8% 

ACSC 

204 

144 

70.  6% 

AW  C 

194 

124 

63.  9% 

Total 

584 

381 

65.  2% 

Demographic  Classification  of  Respondents 


The  381  respondents  represented  a wide  range  of  grade,  aero- 
nautical rating,  command  of  assignment,  years  of  service,  education 
level,  and  other  demographic  variables.  Tables  4.2  through  4.5 
illustrate  respondent  classification  by  grade,  aerona  itical  rating, 
history  as  an  GEn  rater,  and  most  recent  OER  rating.  Additionally, 
16  different  commands  were  represented  in  the  survey  group,  the 
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Table  4.  2 

Exercise  Respondents  by  Grade 


Grade 

Absolute 

Frequency 

Relative 

F requency  (%) 

0-2 

23 

6.  1 

0-3 

107 

28.  1 

0-4 

127 

33.  3 

0-5 

95 

24.  9 

0-6 

29 

7.6 

Total 

381 

100.  0 

Table  4.  3 

Exercise  Respondents  by  Aeronautical  Rating 


Aero 

Absolute 

Relative 

Rating 

F requency 

Frequency  (%) 

Pilot 

164 

43.  0 

Navigator 

49 

12.9 

Non -rated 

168 

44.  1 

Total 

381 

100.  0 

mean  education  level  was  a master's  degree;  and  87%  of  the  respond- 
ents indicated  that  they  either  definitely  or  most  likely  will  make  the 
Air  Force  a career. 


Table  4.4 

Exercise  Respondents  by  History  as  an  OER  Rater 


History  as 
a Rater 

Absolute 

F requency 

Relative 

F requency  (%) 

Never  rated  another 

officer 

117 

30.  7 

Occasionally  rated 
another  officer 

112 

29.4 

Frequency  rated 
another  officer 

150 

39.4 

Missing  Cases 

2 

. 5 

Total 

381 

100.  0 

Table  4.  5 

Exercise  Respondents  by  Most  Recent  OER  Rating 


OER 

Rating 

Absolute 

F requency 

Relative 

F requency  (%) 

II  Jit 

192 

50.4 

"2" 

112 

29,4 

"3"  through  "6"  ' 

71 

18.  6 

Other  or  Never  had 
an  OER 

5 

1.3 

Missing  Cases 

1 

.3 

Total 

381 

100.  0 

^Only  one  officer  had  an  1 

OER  rating  of  "4"  or  less. 
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An  interesting  result  of  one  demographic  question  was  that  31% 
of  the  exercise  respondents  had  never  written  an  OER  on  another 
officer;  this  factor  was  later  used  in  investigating  statistical  differ- 
ences in  regression  models.  Also,  the  distribution  of  the  responses 
to  question  14  ("What  rating  did  you  receive  on  your  most  recent  con- 
trolled OER?")  indicated  that  a majority  of  officers  had  received  "Is" 
(specifically,  50.  5%  versus  a 22%  Air  Force  average).  Furthermore, 
only  18.7%  responded  that  they  had  received  a "3"  or  less,  as  opposed 
to  an  Air  Force  average  of  50%  under  the  1974-1977  OER  system. 
Assuming  valid  responses  to  these  questions,  the  students  at  the  three 
PME  schools  received  higher  OER  ratings  than  Air  Force  officers  in 
general. 

A complete  listing  of  responses  to  the  15  demographic  questions 
is  found  in  Appendix  E. 


Individual  Regression  Results 

As  outlined  in  the  methodology  chapter  of  this  report,  two  regres- 
sion analyses  were  performed  on  the  data  for  each  exercise  respondent 
in  order  to  construct  individual  models  where  the  "degree  of  appropri- 
ateness" of  an  OER  rating  was  the  decision  (dependent)  variable,  and 
the  factors  "performance"  and  "potential"  were  the  predictor  (inde- 
pendent) variables. 

Using  the  previously  mentioned  Fortran  program,  two  regres- 
sion models  were  constructed  for  each  respondent:  one  for  the  nine 
scenarios  with  an  assigned  OER  rating  of  "1,"  and  one  for  the  nine 
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scenarios  with  an  OER  rating  of  "3.  11  The  regression  algorithm  calcu- 
lated the  standardized  regression  coefficients,  relative  weight  for 

2 

each  predictor,  and  squared  multiple  correlation  coefficient  (R  ) for 
each  individual's  pair  of  regression  models. 

In  order  to  more  readily  present  the  result?  £ these  individual 
regressions,  mean  relative  weights  and  mean  1 values  were  calcu- 
lated for  the  groupings  of  officers  by  PME  school  and  are  presented 
in  Table  4.6.  Table  4.7  lists  the  percentages,  by  PME  group,  of  the 
individual  regressions  that  were  significant  at  the  . 05  level. 

A preliminary  inspection  of  the  relative  weights  assigned  gives 
the  appearance  of  general  agreement  among  groups  of  officers  regard- 
ing the  performance  and  potential  criteria  used  in  judging  OER  ratings. 
However,  specific  hypothesis  tests  were  conducted  later  in  the  analysis 
using  group  regressions  to  determine  if  the  group  regression  models 
were  statistically  different. 

It  is  noted  that  "performance"  as  a decision  criterion  consist- 
ently had  a higher  relative  weight  than  "other  indicators  of  potential" 
for  all  of  the  PME  groups.  An  interesting  result  of  the  separate 
regressions  for  "1"  and  "3"  scenarios  was  that  for  all  groups,  the 
relative  weight  for  potential  was  higher  for  the  "3s"  scenarios  model 
than  for  the  "Is"  scenario  model.  This  implies  that  potential  was 
weighted  more  heavily  when  these  officers  made  judgments  of  "bad" 
OERs  than  it  was  when  they  made  judgments  of  "good"  OERs. 
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Table  4.  6 

Individual  Regression  Results 


Respondent 

Group 

Scenario 

OER 

Predictor 

Performance 

OER  = "l" 

Potential 

SOS 

OER  = "3" 

Performance 

Potential 

Performance 

OER="l" 

Potential 

ACSC 

OER="3" 

Performance 

Potential 

Performance 

OER  = "l" 

Potential 

AWC 

OER  = "3  " 

Performance 

Potential 

Performance 

OER  = "l" 

Potential 

All 

Subjects 

OER="3" 

Performance 

Potential 

Mean  Relative 
W eight 


. 70 


Mean 

R2 


. 686 


Mean  individual  R values  were  scattered  about  the  .7  Jevel,  which 

is  not  considered  unusually  low  or  high,  although  other  policy  captur- 

2 

ing  models  have  resulted  in  somewhat  higher  R values  (.  75  and  better) 
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Table  4.  7 

Percentages  of  Individual  Regressions  Significant 

(«  = , 05) 


PME 

Group 

Total  Number 
of  Individual 
Regressions 

OER 

Level 

Number  of 
Regressions 
Significant 
(d  = . 05) 

Percentage 

Significant 

SOS 

i 

113 

1 

85 

75.  2% 

113 

3 

79 

69.  9% 

A CSC 

144 

1 

110 

76.  4% 

144 

3 

110 

76.4% 

AWC 

120 

1 

85 

70.  8% 

120 

3 

86 

71.6% 

Glenn  (1977)  and  O' Berry  (1977)  are  examples  of  such  models,  although 
the  policy  capturing  model  involved  was  not  concerned  with  employee 
appraisal  in  either  case.  Zedeck  and  Kafry  report  that  previous 

2 

studies  of  rater  policies  using  policy  capturing  techniques  yielded  R 
values  ranging  from  .50  to  .80  (Zedeck  and  Kafry,  1977:275). 

One  hypothesis  that  was  tested  by  this  research  was  that  officers 
use  the  same  decision  policies  when  making  judgments  of  both  ''good" 
and  "bad"  OER  ratings,  and  the  data  obtained  from  the  individual 
regression  models  was  used  to  test  this  hypothesis  in  a manner 
described  in  the  next  section  of  this  chapter. 





Paired  Sample  t-Test  of  Individual  Relative  Weights 

As  outlined  in  earlier  discussion,  a paired  sample  t-tesfc  was 
conducted  on  the  relative  weights  that  officers  assigned  (implicitly)  to 
predictors  in  the  two  scenario  OPR  sets  (e.g.,  a ''a"  or  "3"  OER  rat- 
ing). The  null  hypothesis  tested  in  each  case  stated  that  the  mean 
relative  weight  for  each  predictor  was  the  same  for  both  scenario 
OER  levels.  The  alternate  hypothesis  stated  that  the  mean  relative 
weight  for  performance  was  greater  in  the  OER  = l situation.  Rejection 
of  the  null  hypothesis  therefore  indicates  that  officers  used  different 
decision-making  policies  for  judging  "good"  and  "bad"  OER  ratings. 
The  results  of  this  analysis  are  presented  in  Table  4.  8. 


Table  4.  8 

t-Test  of  Individual  Relative  Weights 


Group 

fc0 

l.05“ 

Reject  Hp? 

SOS 

?..  33 

1.  645 

Yes  (p  < .011) 

.A  CSC 

4.  75 

1.  645 

Yes  (p  < .001) 

AWC 

57 

1. 645 

Yes  (p<  . 006) 

All 

Respondents 

5.  65 

1.  645 

Yes  (p<  . 001) 

J 


i 

i 


% 


i 


1 


As  noted  in  the  table,  the  null  hypothesis  that  the  relative  weights  | 

I 

were  the  same,  regardless  of  the  scenario  OER  level,  was  rejected  | 

i 

for  all  groups,  with  significance  levels  ranging  from  .011  to  .001.  5 

X 

3 

This  indicates  that  officers  use  different  decision-making  Doiicies 

1 
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concerning  OER  ratings  which  depend  on  the  particular  OER  rating 


being  judged. 


Group  Regression  Results 


The  analysis  of  group  decision-making  policies  involved  an 


approach  similar  to  that  used  in  the  construction  of  individual  regres- 


sion models,  the  main  difference  being  that  the  group  regressions 
were  based  on  the  combined  decisions  made  by  the  group  members  for 
each  of  the  two  OER  ratings  (e.  g.,  381  X 9 decisions  for  each  OER 
level  if  all  respondents  are  grouped  together). 

The  SPSS  subprogram  "regression"  was  used  in  conducting  group 
regressions:  as  in  the  regression  of  individual  decisions,  the  appro- 
priateness of  assigned  OER  ratings  was  regressed  on  the  two  predictor 
variables  "performance"  and  "potential.  " 

Output  from  each  regression  included  standardized  regression 
coefficients  (beta  weights),  the  significance  level  of  the  model  (all 
models  were  significant  at  the  .001  level),  ANOVA  data  used  in  later 
comparisons  of  regression  models,  and  the  value  for  the  gr^up 
being  modeled.  As  was  pointed  out  in  earlier  discussion,  relative 
weight  is  a concept  that  is  meaningful  only  for  individual  decision- 
makers; thus,  group  regressions  presented  in  this  research  do  not 
have  associated  relative  weights.  Table  4.9  summarizes  the  results 
of  the  primary  group  regressions  that  were  conducted.  In  order  to 
provide  a means  of  comparing  group  and  individual  regression  models, 
both  results  are  listed  in  the  table. 
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Note  that  the  beta  weights  for  the  OER="3"  models  are  negative: 
this  indicates,  as  one  would  expect,  that  the  appropriateness  of  an 
OER  rating  of  "3”  was  judged  to  be  less  as  the  levels  of  performance 
and  potential  increased.  As  in  the  individual  regression  models,  per- 
formance was  weighted  more  heavily  than  potential  in  the  group 
models. 

If  one  examines  the  tabled  R2  values,  it  is  noted  that  group  R2s 

are  somewhat  lower  than  mean  individual  R s.  This  results  from  the 

fact  that  although  each  decision  maker  may  have  made  judgments  with 

a fairly  high  degree  of  consistency,  a model  which  is  based  on  the 

total  number  of  decisions  made  among  all  judges  reflects  a lower 

2 

decision-making  consistency  (R  ) due  to  differences  in  individual 
decision  policies. 

An  interesting  application  of  this  R4,  tendency  in  this  research 
2 

was  that  group  R s for  the  OER="3"  models  were  consistently  lower 

2 7 
than  R s for  the  OER="l"  models,  although  mean  individual  R^  values 

for  the  two  OER  levels  were  similar  in  magnitude.  This  suggests  that 

although  the  consistency  of  individual  judges  was  about  the  same  for 

2 

both  OER  levels,  the  lower  group  R values  for  the  OER="3"  model  is 
indicative  of  more  varied  decision  policies  among  officers  making 
judgments  of  "3"  OER  ratings. 

In  order  to  determine  whether  the  regression  models  for  various 
groups  of  officers  were  different,  F-test  calculations  were  made,  as 
described  in  the  next  section  of  this  chapter. 
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F-Tesfc  Comparir on  of  Croup  .Regression  Models 

The  F-test  calculation  used  to  test  whether  group  regression 
models  being  com,  ..red  were  significantly  different  is  presented  in 
Appendix  D.  In  each  comparison,  the  null  hypothesis  stated  that  there 
was  no  significant  difference  (at  the  . 05  level)  in  the  regression  coef- 
ficients of  the  models  being  compared,  versus  the  alternate  hypothesis 
that  there  was  at  least  one  model  whose  regression  coefficients  were 
different.  The  results  of  the  F-test  comparisons  are  presented  in 
Tables  4.  10  through  4.  13  based  on  groupings  made  by  PME  school, 
aeronautic  a'  rating,  respondents'  history  as  an  OER  rater,  and 
respondents'  most  recent  OER  rating. 


Table  4.  10 

F-Test  Comparison  of  Group  Regression  Models 
(PME  School  Groupings) 


Groups 

Compared 

Scenario 

OER 

* 

N 

F0 

F 

. 05 

Reject  Hq? 

SOS/ACSC/AWC 

II  1 11 

3429 

2.  05 

2.  10 

No 

SOS  /ACSC/AWC 

H3H 

3429 

7.49 

2.  10 

Y es 

SOS/ACSC 

113" 

2313 

.68 

2.60 

No 

SOS/ AW  C 

1.3.1 

2133 

12.  20 

2.  60 

Yes 

ACSC/AWC 

"3" 

2412 

9.  13 

2.60 

Y es 

‘N  = number  of  decisions. 
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Table  4.  11 

F-Test  Comparison  of  Group  Regression  Models 
(Aeronautical  Rating) 


Groups 

Compared 

Scenario 

OER 

N 

Fo 

F.  05 

Reject  HQ? 

Pilot/Navigator  / 

ii  jn 

3429 

. 853 

2.  10 

No 

Non-rated 

Pilot /Navigator  / 

"3" 

3429 

1.  525 

2.  10 

No 

Non-rated 

Table  4.  12 

F-Test  Comparison  of  Group  Regression  Models 
(History  as  an  GER  Rater) 


Groups 

Compared 

Scenario 

OER 

N 

Fo 

F.  05 

Reject  Hq? 

Never/ 

Occasionally/ 

F requently 

II  JM 

3420 

4.  10 

2.  10 

Y es 

Never/ 

Occasionally/ 

F requently 

"3" 

3420 

5.  376 

2.  10 

Yes 

As  illustrated  in  Table  4.  10,  a comparison  of  regression  models 
of  the  three  PME  schools  yielded  interesting  results:  the  null 
hypothesis  was  not  rejected  when  comparing  tho  models  '"ith  an 
assigned  OER  rating  of  "1,  " indicating  that  all  groups  used  similar 
decision  policies  (i,  e.  , used  the  performance  and  potential  criteria 
in  essentially  the  same  manner).  However,  the  null  hypothesis  was 
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Table  4.  13 

F-Test  Comparison  of  Group  Regression  Models"'' 
(Recent  OER  Rating) 


Groupings  by 
Most  Recent 
OER  Rating 

Scenario 

OER  Level 

i 

Fo 

F.  05 

Reject  Hq? 

1/2/3 

"1" 

1.  70 

1.  88 

No 

1/2/3 

"3" 

2.  25 

1.  88 

Yes 

1/2&3 

"3" 

. 162 

2.  6 

No 

1&2/3 

ii  j n 

3.  86 

2.  6 

Yes 

rN  = 3366  Decisions  (374  Respondents) 


rejected  when  comparing  the  models  where  the  assigned  OER  rating 
was  a "3."  This  hypothesis  rejection  led  to  a further  partitioning  of 
the  groups  being  compared:  further  analysis  revealed  that  the  AY/C 
group  had  a decision-making  model  that  was  statistically  different 
from  the  others. 

There  was  no  apparent  difference  in  decision-making  policies 
based  on  aeronautical  rating,  since  the  null  hypothesis  was  not 
rejected. 

When  groupings  were  based  on  the  recent  (controlled)  OER  rating 
of  respondents,  the  results  were  varied;  there  was  no  apparent  differ- 
ence in  judgment  policies  among  these  officers  when  judgments  of  "1" 
ratings  were  made.  However,  when  judgments  of  "3"  ratings  were 
made,  the  null  hypothesis  was  rejected.  Further  analysis  indicated 
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that  the  group  of  officers  having  received  n3s"  on  their  most  recent 
(controlled)  OER  used  significantly  diffeient  decision  policies  when 
judging  the  appropriateness  of  "3"  OER  ratings  in  the  decision-making 
exercis  e. 

These  results  indicate  that  these  officers'  decision-making  poli- 
cies were,  in  some  cases,  significantly  different  on  the  basis  of  (1) 
history  as  an  OER  rater  and  (2)  the  respondent's  recent  OER  rating, 
depending  on  the  OER  rating  being  judged.  When  the  OER  score  being 
judged  was  a "3,  " the  AWC  group  regression  was  significantly  differ- 
ent than  the  models  for  SOS  or  ACSC.  These  differences  do  not  neces 
sarily  suggest  a causal  relationship  between  these  variables  (e.  g. , a 
low  OER  rating  causes  a certain  decision-making  policy).  However, 
by  examining  the  standardized  beta  v.’eignts  listed  in  Tables  4.  14  and 
4.  15,  one  can  observe  the  relative  influerce  that  these  groups  placed 
on  the  performance  and  potential  factors. 


Table  4.  14 

v’. 

Group  Regression  Coefficients 
(History  as  an  OER  Rater) 


Scenario 

OER="  1" 

Scenario 

OER="3" 

Group 

Pi 

(Performance) 

Pi 

(Potential) 

Pi 

(Performance) 

Pi 

(Potential) 

Never  Rated 

Another 

. 664 

. 348 

-.612 

-.  378 

Occasionally 

.659 

. 376 

-.  560 

-.422 

F requently 

.619 

. 383 

-.  545 

-.405 

'‘Standardized 
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Table  4.  15 

Group  Regression  Coefficients* 
(Recent  OER  Rating) 


Group 

OER 

Rating 

Scenario 

OER="  1" 

Scenario 

OER="3" 

4 

(Performance) 

(Potential) 

(Performance) 

^2 

(Potential) 

111  II 

.652 

.384 

-.  573 

-.411 

"2" 

.639 

. 355 

-.  555 

-.427 

M^ll 

.651 

. 342 

-.  606 

-.  345 

Standardized 

Relative  to  groupings  based  on  one's  history  as  an  OER  rater,  the 
analysis  indicates  that  officers  who  have  more  frequently  rated  others 
w eight  potential  most  heavily.  This  is  probably  true  because  there  is 
a high  correlation  between  frequency  as  a rater  and  an  officer's 
grade,  coupled  with  the  fact  that  the  AWC  group  (senior  officers) 
weighted  potential  most  heavily. 

When  groupings  were  made  by  the  recent  (controlled)  OER  rating 
of  respondents,  the  beta  weights  indicate  that  those  officers  having 
received  the  lowest  OER  ratings  ("3s")  weighted  performance  most 
heavily,  for  the  scenario  OER  = "3"  model. 

In  summary,  group  regressions  indicated  that  the  AWC  group 
weighted  potential  most  heavily  (compared  to  the  other  PME  groups). 
Additionally,  officers  who  have  frequently  been  OER  raters  also 
weighted  potential  most  heavily.  However,  officers  who  received 
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recent  OER  ratings  of  "3"  weighted  performance  most  heavily,  and 
groupings  based  on  aeronautical  rating  did  not  result  in  significantly 
different  weightings  among  officers. 


Examination  of  an  Interactive  Model 

As  explained  in  earlier  discussion,  this  research  was  primarily 
concerned  with  judgment  modeling  based  on  an  additive  effect  of  pre- 
dictor variables.  However,  an  interaction  term  was  added  to  the 
group  regression  models  in  order  to  examine  the  "strength"  of  the 
interaction  between  performance  and  potential  predictors.  The  inter- 
action term  (the  performance  level  times  the  potential  level)  was 
added  to  each  regression  model  in  the  final  regression  step,  and  the 
significance  and  R^  increases  were  noted.  The  results  of  this  analysis 
are  presented  in  Table  4.  16,  as  compared  with  the  non-interactive 
model. 


The  significance  level  of  the  interaction  term  was  . 001  in  all 
cases,  although  the  R increases  appeared  to  vary  widely  depending 
on  the  ocenario  OER  level.  For  ali  groups,  the  interaction  term 
added  more  explanative  power  to  the  regression  models  involving 
"good"  OER  ratings  than  it  did  to  the  models  involving  "bad"  ratings. 
In  essence,  there  appears  to  have  been  greater  interaction  between 
performance  and  potential  criteria  when  decisions  about  "good"  OERs 
were  made.  Since  the  R“  values  for  the  OER-"3"  models  did  not 
increase  appreciably  when  the  interaction  term  was  added,  it  appears 


Group 

Scenario 

OER 

Group 

(No  Interaction) 

Group  R^ 

(With  Interaction) 

% Increase 
in  RZ 

SOS 

1 

. 555 

. 6 1 6 

11.  0% 

3 

.519 

. 526 

1.  3% 

A CSC 

1 

.613 

. 673 

9.  8% 

3 

. 519 

. 529 

.2% 

- - j 

AW  C 

1 

.486 

. 553 

13.8% 

3 

.432 

.438 

1.4% 

All 

Respond- 

1 

' 

. 552 

.615 

11.4% 

ents 


3 


.486 


.489 


.6% 
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Table  4.1/ 

Part  III  Responses 

(Mean  Values  and  Standard  Deviation) 


Situation 


Give 


M- 


Absolute 


Relative 


Promotion 

Potential 

Effective- 
ness at 
Higher 
Levels 


4.72 


4.67 


a 

lx 

.51 

4.  69 

.63 

4.  59 

. 58 

4.67 

. 54 

4.49 

Receive 

i a "3" 

ll 

o 

4.  00 

1.  08 

4.  20 

.97 

3.  87 

1.  18 

3.  86 

1.  12 

It  is  noted  that  the  mean  scores  for  "performance"  beli.  is  were 
generally  higher  than  the  mean  scores  for  "potential"  beliefs.  In 
order  to  test  whether  the  performance  scores  were  statistically  differ- 
ent from  the  potential  scores,  a paired  sample  t-test  was  conducted  on 
the  mean  values  of  aggregate  performance  and  potential  scores  for 
each  OER  situation. 

For  each  OER  situation  the  agreement  scores  for  the  two  per- 
formance beliefs  were  added  and  compared  to  a similarly  constructed 
score  for  the  two  potential  beliefs.  The  t-test  tested  the  null 
hypothesis  that  the  means  of  the  two  scores  were  the  same,  versus 
the  alternate  hypothesis  that  the  performance  score  was  higuer.  The 
results  of  these  t-tests  are  presented  in  Table  4.  18. 


Table  4.  18 

o. 

Paired  Sample  t-Test'1' 
(Part  III  Data) 


In  most  cases,  the  null  hypothesis  was  rejected;  however,  in  the 
situation  where  an  individual  has  (hypothetically)  given  a "1,  11  the  null 
hypothesis  could  not  be  rejected. 

The  general  indication  of  this  analysis  is  that  this  stronger 
agreement  with  performance  beliefs  is  consistent  with  the  findings  of 
the  policy  capturing  model,  although  there  was  strong  agreement  with 
the  stated  beliefs  in  all  situations  and  therefore  not  much  variance  in 
the  Part  III  responses. 


Situation 

fco 

fc.05 

Reject  Hq? 

Probability 

% 

Give  a "1" 

.96 

1. 645 

No 

p < . 169 

A 

Receive  a "1" 

2.  30 

1.  645 

Yes 

p < .011 

j 

Give  a '‘3" 

13.  89 

1.  645 

Yes 

p<  .001 

j 

Receive  a "3M 

6.  19 

1.645 

Y es 

p < .001 

*N  = 377 

79 


in : ayy* 


J 


V Summary  and  Conclusions 

The  managerial  activity  known  as  employee  appraisal  has 
historically  been  viewed  by  many  as  a hindrance  rather  than  an  aid  to 
organizational  goal  accomplishment.  This  negative  view  of  appraisal 
systems  may  result  from  many  factors,  one  of  which  is  the  subjec- 
tivity inherent  in  a system  in  which  an  individual's  past  and/or  future 
worth  to  an  organization  is  evaluated  by  another  individual  (or  indi- 
viduals). 

The  primary  objective  of  employee  appraisal  is  to  improve 
results,  i.  e.  facilitate  organizational  goal  accomplishment.  Many 
appraisal  systems  are  structured  such  that  the  sole  emphasis  is  on 
the  appraisal  of  past  accomplishments  during  a rating  period,  i.  e. 
performance  appraisal.  However,  some  managerial  appraisal  sys- 
tems are  designed  to  evaluate  individual  potential  in  addition  to  one's 
performing  during  a reporting  period. 

The  evaluation  of  potential  involves  an  estimation  of  an  individ- 
ual's future  worth  to  an  organization,  which  subsequently  forms  an 
information  base  upon  which  job  assignment,  promotion,  career  pro- 
gression, and  other  personnel  administration  decisions  are  based. 
Since  potential  evaluation  is  future -oriented,  the  criteria  used  in  mak- 
ing such  assessments  of  managers  appear  to  result  from  individual 
perceptions  of  what  factors  should  be  considered.  There  is  general 
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agreement  that  one's  level  of  potential  is  indicated  in  part  by  past  :* 

performance;  however,  potential  assessment  theoretically  involves 
more  than  a de  facto  appraisal  of  performance.  The  appraisal  of 
potential  generally  involves  an  examination  of  trends  in  past  perform- 
ance, coupled  with  an  assessment  of  personality  traits  and  individual 
capabilities;  the  frame  of  reference  is  generally  with  respect  to  some 
future  type  of  job  of  expanded  and/or  different  responsibilities. 

Due  to  the  interrelated  nature  of  performance  and  potential,  as 
well  as  the  difficulties  and  varied  perceptions  involved  in  assessing 
these  two  concepts,  there  is  an  everpresent  possibility  that  individu- 
als' concepts  of  performance  and  potential  could  become  confused  and 
overlap  significantly  when  judgments  of  appraisal  ratings  are  made, 
i.  e. , potential  is  perceived  to  be  heavily  (if  not  totally)  determined  by 
one's  performance  during  a rating  period.  If  the  assessment  of 
potential  is  based  on  widely  varied  perceptions  (among  those  involved 

with  the  appraisal  system)  of  the  criteria  that  should  be  used,  then  j 

| 

organizational  dysfunctions  may  result.  More  specifically,  these 
varied  perceptions  among  raters  and  ratees  may  lead  to  conflict  or 
frustrations  which  adversely  affect  the  subordinate's  performance, 
and  henc^'  organizational  goal  accomplishment. 

The  Air  Force  officer  evaluation  system  is  one  appraisal  system 
that  requires  an  evaluation  of  potential,  and  the  process  is  therefore 
subject  to  officers'  varied  perceptions  of  what  criteria  should  be  used 
to  rate  potential.  The  degree  to  which  factors  other  than  past 


performance  determine  this  rating  of  potential  is  not  outlined  in  Air 
Force  policy,  hence  the  decision  is  made  by  each  individual  officer. 

This  research  effort  involved  an  examination  of  the  relative 
influence  that  selected  performance  and  potential  criteria  have  on 
officers'  judgments  of  CER  ratings,  particularly  since  some  writers 
have  suggested  that  the  evaluation  of  potential  may  be  little  more  than 
an  evaluation  of  past  performance.  Since  the  judgment  policies  of  Ail- 
Force  officers  are  a result  of  individual  perceptions,  modeling  these 
judgment  policies  was  the  primary  method  used  in  this  research  for 
determining  the  influence  of  performance  and  potential  criteria  on  the 
judgment  of  OER  ratings. 

The  methodology  used  in  this  research  effort  involved  what  is 
known  as  judgment  modeling,  or  "policy  capturing.  " Since  the  officer 
evaluation  system  involves  judgments  of  officers'  potential,  the  policy 
capturing  technique  was  employed  to  construct  a mathematical  model 
of  the  process  by  which  these  judgments  were  made.  The  regression 
models  constructed  in  this  research  were  primarily  based  on  the 
assumption  that  human  judgment,  although  itself  a very  complex 
process,  can  be  effectively  modeled  as  a linear  combination  of  stimu- 
lus variables,  or  cues.  In  this  study,  the  decision  variable  was  the 
"degree  of  appropriateness"  of  an  OER  rating  assigned  to  a hypothetical 
officer,  and  the  two  predictor  variables  were  indicators  of  perform- 
ance and  potential.  Respondents  to  the  decision-making  exercise  were 
asked  to  judge  the  appropriateness  of  OER  ratings  given  to  18 
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hypothetical  officers,  each  of  whom  was  described  in  terms  of  differ- 
ent levels  of  performance  and  potential  attributes.  The  exercises  that 
were  returned  by  381  PME  school  students  were  used  to  construct  both 
individual  and  group  regression  models  which  described  the  decision- 
making process  of  Air  Force  officers  in  making  judgments  of  OER 
ratings. 


Conclusions 

The  various  analysis  results  presented  in  chapter  four  of  this 
report  are  related  at  this  point  to  the  stated  research  hypotheses.  In 
each  case,  it  is  noted  whether  or  not  the  null  hypothesis  was  rejected, 
and  the  data  analysis  which  supports  the  rejection  (or  non- rejection) 
is  identified. 

HI:  Air  Force  officers  combined  performance  and  potential 
criteria  in  an  additive  rather  than  interactive  manner  when 
making  judgments  of  OER  ratings. 

Group  regression  models  which  included  an  interaction  term 
(performance  level  times  the  potential  level)  resulted  in  statistically 
significant  interaction  terms,  although  the  percentage  increase  in  R*% 
i.  e. , the  added  explanative  power  of  the  model,  was  not  considered 
substantial  enough  to  question  the  value  of  the  linear  models  in 
describing  officers'  judgment  policies.  The  interaction  term  did  lead 
to  higher  percentage  increases  in  R^  values  for  the  OER="l"  models 
than  the  OER  = "3"  models;  however,  the  fact  that  non-interactive 


models  accounted  for  86.  2%  to  99.  8%  of  the  explained  variance  in  the 





H2:  Air  Force  officers,  -when  considering  (1)  job  factors 
which  are  indicators  of  immediate  past  performance 
and  (2)  persona]  factors  which  are  (other)  indicators  of 
potential,  will  make  judgments  of  OER  ratings  (Section 
IV,  "Evaluation  of  Potential")  based  equally  on  the  job 
and  personal  factors. 

For  the  performance  and  potential  factors  chosen,  the  group 

regression  results  presented  in  Table  4.  8 show  that  the  ratio  of  the 

beta  weight  for  performance  to  the  beta  weight  for  potential  ranged 

from  a high  of  1.  83  to  a low  of  1.  32.  Additionally,  the  mean  relative 

weights  for  the  two  predictors  that  were  calculated  from  individual 

regression  models  indicated  a fairly  strong  emphasis  on  performance: 

mean  relative  weights  ranged  from  .73  (performance)  and  .27 

(potential)  to  .60  (performance)  and  .40  (potential).  These  analysis 

results  therefore  lead  to  the  rejection  of  hypothesis  H2;  i.  e. , officers 

place  greater  weight  on  the  performance  factor. 

H3:  Air  Force  officers  use  the  same  judgment  policies 
when  judging  either  "good"  or  "bad"  OER  ratings  'i.  e.  , 

"1"  and  "3.  " respectively)  in  terms  of  the  performance 
and  potential  factors  that  influence  these  judgments. 

This  hypothesis  was  rejected,  based  on  the  results  of  the  paired- 
sample  t-test  on  the  relative  weights  calculated  in  the  individual 
regression  models.  This  rejection  indicates  that  officers  use  signifi- 
cantly different  policies  when  judging  good  and  bad  OER  ratings. 
Specifically,  potential  was  weighted  more  heavily  when  officers  made 
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rater  policies,  and  therefore  supports  hypothesis  H3.  It  is  noted, 
however,  that  the  mean  R"  values  for  the  AWC  students  were  the  low- 
est of  all  the  groups  surveyed,  particularly  for  the  scenario  OER 
equal  to  "3"  models. 

H5:  Air  Force  officers  represent  a homogeneous  group 
of  decision  makers  regarding  OER  scores;  i.  e. , use  the 
same  decision  policies  when  making  judgments  of  OER 
ratings,  regardless  of  data  groupings  by  PME  school, 
aeronautical  rating,  OER  rating  history,  and  whether  or 
not  officers  have  been  OER  raters. 

Based  on  the  group  regression  models  for  the  scenario  OER  of 
"1,"  this  hypothesis  was  not  rejected  when  comparisons  were  made  on 
the  basis  of  PME  school  or  aeronautical  rating,  but  the  hypothesis  was 
rejected  when  groupings  were  made  on  the  l s of  respondents' 
history  as  a rater. 

For  the  group  regression  models  with  the  scenario  OER  of  "3," 
the  results  were  different.  The  hypothesis  was  rejected  for  all  group- 
ings of  the  respondents  with  the  exception  of  aeronautical  rating. 
Further  analysis  of  the  groupings  by  PME  school  indicated  that  the 
AWC  group  had  a statistically  different  regression  model  than  the 
other  two  PME  groups  for  the  OER="3"  model.  Note  that  the  rejec- 
tion of  this  hypothesis  indicates  that  there  was  variability  (sometimes 
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substantial)  among  officers'  judgment  policies;  in  cases  where  vari- 
ability was  great,  this  may  be  an  indication  of  some  confusion  (or  at 
least  lack  of  agreement)  among  officers  regarding  what  constitutes  a 
rating  of  "potential.  " 

H6:  The  relative  weights  for  performance  and  potential 
criteria  determined  by  the  policy  capturing  model  are 
consistent  with  the  intuitive  responses  given  in  Part  III 
of  the  exercise;  i.  e.  , modeled  judgment  policies  are  con- 
sistent with  officers'  intuitive  reactions. 

This  hypothesis  is  supported  by  the  results  of  the  paired  sample 
t-test  of  the  performance  versus  potential  "agreement  scores"  pre- 
sented in  Table  4.  18.  It  is  noted,  however  that  this  particular  analysis 
was  diluted  somewhat  by  the  strong  agreement  r.hown  by  respondents 
with  all  reactions  presented  in  the  exercise  (see  Table  4.  17). 

Implications  of  the  Study 

As  can  be  seen  from  the  results  of  this  research  and  the  con- 
clusions drawn,  the  judgment  policies  of  Air  Force  officers  regarding 
OER  ratings  are  quite  varied:  for  all  groups,  the  models  were  differ- 
ent depending  on  whether  the  OER  ratings  assigned  were  "good"  or 
"bad.  " Additionally,  although  the  consistency  with  which  officers 
made  judgments  of  the  ratings  assigned  was  considered  average  for 
this  type  of  research,  the  mean  individual  values  for  AWC  students 

(Lieutenant  Colonels  and  Colonels)  were  the  lowest  of  all  groups  sur- 

2 

veyed.  Furthermore,  the  low  group  R values  (compared  to  the  other 
PME  groups)  indicated  that  there  was  a wider  variety  in  the 
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decision-making  policies  of  AWC  students;  i.  e.,  for  this  exercise  the 
AWC  group  was  not  a very  homogeneous  group  of  decision  makers. 
Further  support  for  the  differences  in  the  AWC  group  was  provided  by 
the  F-tests  on  the  group  regressions,  which  singled  out  AWC  as  having 
significantly  different  regression  models  than  the  other  two  schools 
(the  AWC  group  weighted  potential  more  heavily  for  the  model  where 
the  scenario  OER  rating  was  a "3"). 

A Final  Comment 

The  original  statement  of  the  research  problem  indicated  that 
there  may  be  confusion  among  officers  regarding  the  concepts  of  per- 
formance and  potential.  In  an  attempt  to  address  this  problem,  one 
area  that  was  investigated  was  the  relative  influence  that  performance 
and  potential  (as  represented  by  three  indicators  each)  had  on  officers' 
judgments  of  OER  ratings  (which  are,  by  definition,  ratings  of  poten- 
tial). It  is  clear  that  there  is  no  correct  answer  to  the  question  of 
what  emphasis  should  be  placed  on  performance  versus  ether  indi- 
cators of  potential;  since  Air  Force  policy  does  not  address  the  issue, 
each  individual  officer  makes  judgments  of  OER  ratings  based  on  his 
or  her  perceptions  of  what  criteria  should  be  used  to  evaluate  poten- 
tial. This  situation  therefore  assures  a great  amount  of  subjectivity 
in  an  appraisal  process  that  is  considered  by  many  to  be  the  critical 
determinant  of  every  officer's  future  in  the  Air  Force. 

A more  definitive  indicator  of  this  confusion  between  perform- 
ance and  potential  was  the  variability  among  officers'  judgment 
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policies  regarding  OER  ratings:  the  rejection  for  some  groups  of 
hypothesis  H5,  which  related  to  group  decision-making  homogeneity, 
is  an  indication  of  some  confusion  among  officers  regarding  what 
constitutes  "potential.  11 
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A DECISION  MAKING  EXERCISE 
FOR 

AIR  FORCE  OFFICERS 


THIS  IS  NOT  A QUESTIONNAIRE.  It  is  a decision  making 
ex  rcise  that  is  designed  to  determine  how  Air  Force 
officers  make  decisions  regarding  employee  appraisal 
systems.  The  information  derived  from  this  exercise 
will  be  used  in  a master's  thesis  effort  by  one  of  your 
fellow  officers  at  the  Air  Force  Institute  of  Technology. 

Please  cooperate  in  caiefully  completing  all  three  parts 
of  this  exercise  so  that  we  may  gain  additional  insights 
into  the  decision  making  patterns  of  Air  Force  officers. 
Your  participation  in  this  exercise  will  be  kept  strictly 
conf idantial . 

Thank  you! 


PLEASE  DO  NOT  PUT  YOUR  NAME  ON  THIS  EXERCISE 


U3AF  SCN  78-108 
(Expires  23  Sep  ?8) 
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PRIVACY  STATEMENT 


In  accordance  with  paragraph  30,  AFR  12-35 > the  following 
information  is  provided  as  required  by  the  Privacy  Act  of  197^-: 

a . Authority 

(1)  5 U.S.C  301 , Departmental  Regulations ; and/or 

(2)  10  U.S.C.  80-12,  Secretary  of  the  Air  Force.  Powers 
and  Duties.  Delegation  by. 

b.  Principal  purposes.  The  survey  is  being  conducted  to 
collect  information  to  be  used  in  research  aimed  at  illuminating 
and  providing  inputs  to  the  solution  of  problems  of  interest 

to  the  Air  Force  and/or  DOD. 

c.  Routine  Uses.  The  survey  data  will  be  converted  to 
information  for  use  in  research  of  management  related  problems. 
Results  of  the  research, based  on  the  data  provided,  will  be  in- 
cluded in  written  master's  thesis  and  may  also  be  included  in 
published  articles,  reports,  or  texts.  Distribution  of  the 
results  of  the  research,  based  on  the  survey  data,  whether  in 
written  form  or  orally  presented,  will  be  unlimited. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  No  adverse  action  of  any  kind  may  be  taken  against 
any  individual  who  elects  not  to  participate  in  any  or  all  of 
this  survey. 


Part  I : 


BACKGROUND  INFORMATION 


Please  circle  the  letter  corresponding  to  the  most  appropriate 
answer  for  each  of  the  following  questions: 


1 .  What  is  your  present  grade? 


cl  * 

Second  Lieutenant 

d. 

Major 

b. 

First  Lieutenant 

e. 

Lieutenant  Colonel 

c. 

Captain 

f . 

Colonel 

2.  What  aeronautical  rating  do  you  hold? 

a.  Pilot  c.  Flight  Surgeon  e.  Other 

b.  Navigator  d.  Non-rated 


3. 


To  what  command  are  you  currently  assigned? 


a.  Alaskan  Air  Command 

b.  U.S.  Air  Force  Academy 

c.  Aerospace  Defense  Command 

d.  U.S.  Air  Forces  in  Europe 

e.  Air  Force  Accounting  and 
Finance  Center 

f.  Air  Force  Logistics  Command 

g.  Air  Force  Systems  Command 

h.  Air  Reserve  Personnel  Center 

i.  Air  Training  Command  (now 
includes  Air  University) 

j.  Headquarters  Air  Force  Reserve 

k.  Headquarters  USAF 

l.  Air  Force  Communications 
Service 


m.  Air  Force  Data  Automation  Agency 

n.  Headquarters  Command 

o.  Military  Airlift  Command 

p.  Pacific  Air  Forces 

q.  Strategic  Air  Command 

r.  Tactical  Air  Command 

s.  USAF  Security  Service 

t.  Air  Force  Military  Personnel  Center 

u.  Air  Force  Inspection  and 
Safety  Center 

v.  Air  Force  Audit  Agency 

w.  Air  Force  Office  of 
Special  Investigations 

x.  Other  ( specify  ) 


4. 


What  is  your  highest  level  of  education  now? 


a. 

Some  high  school  (did  not  graduate) 

e. 

b. 

High  school  graduate  (no  college) 

f. 

c. 

Some  college  (no  degree) 

d. 

College  degree 

S- 

h. 

Graduate  work  (no  master's 
degree) 

Master's  degree 

Postgraduate  work  beyond 
master's  (no  doctorate) 

Doctorate  degree 
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5.  How  much  total  active  federal  military  service  have  you  completed? 


a. 

Less  than  i year 

1. 

Between  11  and  12  years 

b. 

Between  1 and  2 years 

m. 

Between  12  and  13  years 

c. 

Between  2 and  3 years 

n. 

Between  13  and  14  years 

d. 

Between  3 and  4 years 

0. 

Between  14  and  15  years 

e. 

Between  4 and  5 years 

P* 

Between  15  and  16  years 

f. 

Between  5 and  6 years 

0.  * 

Between  16  and  17  years 

g* 

Between  6 and  7 years 

r. 

Between  17  and  18  years 

h. 

Between  7 and  8 years 

s. 

Between  18  and  19  years 

i. 

Between  8 and  9 years 

t. 

Between  19  and  20  years 

j* 

Between  9 and  10  years 

u. 

20  years  or  more. 

k. 

Between  10  and  11  years 

6.  Which  one  of  the  following  do  you  consider  yourself? 

a.  American  Indian 

b.  Asian  Origin 

c . Black 

d.  Spanish  Speaking  Origin 

e.  White  (other  than  Spanish  Speaking  Origin) 

f.  Other  (specify ) 


7.  Which  one  of  the  following  best  describes  your  attitude  toward 
making  the  Air  Force  a career? 

a.  Definitely  intend  to  make  the  Air  Force  a career 

b.  Most  likely  will  make  the  Air  Force  a career 

c.  Undecided 

d.  Most  likely  will  not  make  the  Air  Force  a career 

e.  Definitely  do  not  intend  to  make  the  Air  Force  a career 

THE  NEXT  FOUR  QUESTIONS  ARE  CONCERNED  WITH  YOUR  ATTITUDES  TOWARD 
YOUR  CURRENT  AIR  FORCE  JOB.  IF  YOU  ARE  CURRENTLY  A STUDENT,  PLEASE 
ANSWER  THESE  QUESTIONS  IN  THE  CONTEXT  OF  YOUR  LAST  NON-STUDENT  JOB. 

8.  Choose  the  statement  which  best  describes  how  well  you  like  your  job. 


a. 

I hate  it 

e. 

I 

like  it 

b. 

I dislike  it 

f. 

I 

am  enthusiastic  about  it 

c. 

I don't  like  it 

g- 

I 

love  it 

d. 

I am  indifferent  to  it 

9.  YFhich  one  of  the  following  shows  how  much  of  the  time  you  feel 


satisfied  with  your  job? 

a. 

All  of  the  time 

e. 

Occasionally 

b. 

Most  of  the  time 

f. 

Seldom 

c. 

A good  deal  of  the  time 

g* 

Never 

d. 

About  half  of  the  time 

10.  Which  one  of  the  following  shows  how  you  think  you  compare  with 
other  people? 

a.  No  one  likes  their  job  better  than  I like  mine 

b.  I like  my  job  much  better  than  most  people  like  theirs 

c.  I like  my  job  better  than  most  people  like  theirs 

d.  I like  my  job  about  as  well  as  most  people  like  theirs 

e.  I dislike  my  job  more  than  most  people  dislike  theirs 

f.  I dislike  my  job  much  more  than  most  people  dislike  theirs 

g.  No  one  dislikes  their  job  more  than  I dislike  mine 

11.  Which  one  of  the  following  best  describes  how  you  feel  about 
changing  your  job? 

a.  I would  quit  this  job  at  once  if  I could 

b.  I would  take  almost  any  other  job  in  which  I could  earn 
as  much  as  I am  earning  now 

c.  I would  like  to  change  both  my  job  and  my  occupation 

d.  I would  like  to  exchange  my  present  job  for  another  one 

e.  I am  not  eager  to  change  my  job,  but  I would  do  so  if 
I could  get  a better  job 

f.  I cannot  think  of  any  jobs  for  which  I would  exchange 

g.  I would  not  exchange  my  job  for  any  other 

12.  Which  statement  best  describes  your  involvement  as  an  OER  rater? 

a.  I have  never  written  an  OER  on  another  officer 

b.  I have  occasionally  written  OER's  on  other  officers 

c.  I have  frequently  written  OER's  on  other  officers  (i.e.,  usually 
supervise  at  least  one  other  officer) 
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13.  Do  you  currently  supervise  and  subsequently  write  OER's  on  at 
least  one  other  officer?  (If  you  are  currently  a student, 
please  answer  in  the  context  of  your  most  recent  non-student 
assignment) 

a.  Yes 

b.  No 

14.  What  rating  did  you  receive  on  your  most  recent  controlled  OER 
(reviewer's  rating)  ? 

a.  "1"  c.  "3"  e.  Other  (specify ) 

b.  "2"  d.  "4,”  "5i"or  "6"  f . I have  never  received  a 

controlled  OER 

15.  What  rating  did  you  receive  on  your  second  mos-.  recent  controlled 
OER  (reviewer's  rating)  ? 

a.  "1"  c.  "3"  e.  Other  (specify ) 

b.  "2"  d.  "4,"  "5, "or  "6"  f.  I have  received  only 

one  controlled  OER 


PIEASE  CONTINUE  WITH  PART  II  ON  NEXT  PAGE 
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PAST  II  : EXERCISE  INSTRUCTIONS 


On  the  following  pages  of  this  exercise  are  a number  of  scenarios 
which  present  a rater 1 s observations  of  a subordinate  officer  currently- 
being  evaluated  through  the  OER  system.  These  observations  are  listed 
in  two  categories:  Job  Factors  and  Personal  Factors.  In  some  cases, 
no  observations  will  be  given  in  one  or  both  categories.  In  these  cases, 
assume  that  there  were  no  distinguishing  observations,  either  good  or 
bad,  related  to  the  category(s)  that  was (were)  omitted.  Also  assume  that 
the  Job  and  Personal  Factors  noted  in  each  scenario  are  observations  by 
the  rater  and  have  not  necessarily  been  included  as  written  statements  on 
the  OER  form  itself. 

Following  the  listing  of  rater  observations,  a statement  is  made 
which  identifies  tne  OER  rating  that  was  assigned  to  the  officer  being 
rated.  For  the  purpose  of  this  exercise,  assume  that  there  was  agree- 
ment among  the  rater,  additional  rater,  and  reviewer  regarding  the  rating 
assigned. 

INTERPRET  ALL  RATINGS  IN  THE  CONTEXT  OF  THE  MOST  RECENT  OER  SYSTEM: 

(1)  A "1"  rating  is  controlled,  with  a maximum  of  22% 
of  officers  being  rated  receiving  such  a "top  block 
rating. 

(2)  "2's"  through  "S's”  are  not  controlled. 

For  the  purpose  of  this  exercise,  assume  that  the  following  rating 
percentages  apply  to  all  scenarios: 

OER  Rating  Percentage  of  Officers  Receiving  This 

Rating  in  This  Review  Group 


1 22% 

2 58% 

3 20% 

Remaining  Blocks  8% 

(4  through  6) 


Therefore,  a "1"  rating  indicates  that  at  least  ?8$  of  the  officers 
being  rated  were  rated  below  the  ratee  who  received  a "1."  Similarly, 
a "3"  rating  means  that  at  least  80%  of  the  officers  being  rated  were 
rated  above  rhe  ratee  who  received  a "3." 

After  the  scenario  has  been  presented,  you  are  then  asked  to 
evaluate  each  situation  based  only  on  the  information  provided  in 
the  scenario  and  your  perception  of  whether  or  not  the  OER  rating 
assigned  is  appropriate  for  the  situation  described.  Assume  that 
none  of  the  officers  in  these  scenarios  deserve  an  OER  rating  lower 
than  a "3«"  ~ 

ALWAYS  ASSUME  THAT  THE  RATEE' S GRADE  IS  THE  SAME  AS  YOUR  OWN! 
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EXAMPLE  . SCENARIO 


Each  case  is  presented  to  you  in  the  following  formats 


OFFICER  # 0 


A.  JOB  FACTORS: 

* (either  three  observations  are  listed,  or  the  statement 

2.  "No  distinguishing  observations,  either  good  or  bad,  are  noted") 

3. 

B.  PERSONAL  FACTORS: 

1. 

(either  three  observations  are  listed,  or  the  statement 

"No  distinguishing  observations,  either  good  or  bad,  are  noted") 

3. 

C.  An  OER  rating  of  was  given  (either  a "1"  or  "3"  will  be  entered) 

DECISION  # 0 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


PIEASE  COMPLETE  ALL  CASES.  AS  ALL  CASES  ARE  DIFFERENT 
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OFFICER  #i 

A.  JOB  FACTORS: 

1.  Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "I"  was  given. 


DECISION  # 1 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


*^^****^*^**^^*****-****-x-**  x x x xx  x********-******************************* 

OFFICER  # 2 

A.  JOB  FACTORS: 

1.  Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  PERSONAL  FACTORS: 

1.  Not  very  dedicated  or  enthusiastic. 

2.  Of  average  intelligence. 

3.  Not  very  personable  or  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "3"  was  given. 


‘i 

i 


DECISION  # 2 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 .4 5 

Definitely  NOT  Frobably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


OFFICER  # 3 


A.  JOB  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

B.  PERSONAL  FACTORS: 

1.  Not  very  dedicated  or  enthusiastic. 

2.  Of  average  intelligence. 

3.  Not  very  personable  or  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "1"  was  given. 

DECISION  # 3 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


OFFICER  § 4 

A.  JOB  FACTORS: 

1 . Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # 4 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


OFFICER  # 5 


A.  JOB  FACTORS: 

1.  Accomplished  all  assignments. 

2.  Efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  very  well  done. 

B.  PERSONAL  FACTORS: 

1.  Not  very  dedicated  or  enthusiastic. 

2.  Of  average  intelligence. 

3.  Not  very  personable  or  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "1"  was  given. 


DECISION  # 5 

How  appropriate  is  the  OER  rating  that  was  given? (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


*X  X ***********  lli********#**#***~A  ***x-x-x-x-*************x-*********x-**-x- X********* ** 

OFFICER  # 6 


A.  JOB  FACTORS: 

1.  Accomplished  all  assignments. 

2.  Efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  very  well  done. 

B.  PERSONAL  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # 6 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 


1 

5 • 

Definitely  NOT 

Probably  NOT 

Cannot 

Probably 

Definitely 

Appropriate 

Appropriate 

Determine 

Appropriate 

Appropriate 
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OFFICER  # 7 


A.  JOB  FACTORS*. 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

B.  -PERSONA!  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # 7 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

i 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 

OFFICER  # 8 

A.  JOB  FACTORS: 

1.  Accomplished  all  assignments. 

2.  Efficient  In  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  very  well  done. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # 8 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 
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OFFICER  # 9 


A.  JOB  FACTORS! 

1.  Did  not  accomplish  all  assignments . 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  PERSONAL  FACTORS: 


No  distinguishing  observations,  either  good  or  bad,  are  noted. 


G.  An  OER  rating  of  "3"  was  given. 


DECISION  # 9 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

i 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


*W*JHHt****M*****-X*********-**-X****yi  X X*X  XX-tt***-X-*****»^***3HHfr****^********** 

OFFICER  #10 


A.  JOB  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # ID 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 
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OFFICER  # 11 

A.  JOB  FACTO RSi 

1.  Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement. 

B.  -PERSONAL  FACTORS! 

1.  Not  very  dedicated  or  enthusiastic. 

2.  Of  average  intelligence. 

3.  Not  very  personable  or  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "1"  was  given. 

DECISION  # 11 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 ...3 .....4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 

*W******^W***********r* X**X  X Xttfrtttt^Wfr^^***^******^**^*^  *****-***** ****** 

OFFICER  #12 

A.  JOB  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

B.  PERSONAL  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

C.  An  OER  rating  of  "1"  was  given. 

DECISION  # 12 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2... 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 
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OFFICER  #13 


A.  JOB  FACTORS! 

1.  Did  not  accomplish  all  assignments. 

2.  Not  efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  needed  improvement.  | 

\ 

B.  PERSONAL  FACTORS:  1 


No  distinguishing  observations,  either  good  or  bad,  are  noted. 


C.  An  OER  rating  of  "1"  was  given.  : 

DECISION  # 13  \ 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number)  \ 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 

3 

-X-X  X X X X*******^**-*^*********-**-**  X-X-X  X'  x^***********^******^**^****-*-**********-*  '• 

OFFICER  # 14 

A.  JOB  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 


B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic. 

2.  Highly  intelligent, 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "1"  was  given. 

DECISION  # 14 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1., 2 ...3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


1 08 


1 ik  1.  v'jLu  AAaWA. 


B.  PERSONAL  FACTORS: 

1.  Not  very  dedicated  or  enthusiastic. 

2.  Of  average  intelligence. 

3.  Not  very  personable  or  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "3"  was  given. 

DECISION  # 16 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannot  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 
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OFFICER  #17 


A.  JOB  FACTORS! 

1.  Accomplished  all  assignments. 

2.  Efficient  in  the  use  cf  time  and  other  resources. 

3.  On-the-job  communications  very  well  done. 

B.  PERSONAL  FACTORS: 

1.  Very  dedicated  and  enthusiastic, 

2.  Highly  intelligent. 

3.  Very  personable  and  sensitive  to  people  and  their  needs. 

C.  An  OER  rating  of  "1"  was  given. 


DECISION  #1? 

How  appropriate  is  the  OER  rating  that  wos  given?  (circle  one  number) 


1 


2 


3 


4 


5 


Definitely  NOT 
Appropriate 


Probably  NOT 
Appropriate 


Cannot 

Determine 


Probably  Definitely 

Appropriate  Appropriate 


e. 

| ****-**->( ■****iH!-*********-*'X-******^  *•*•** XX  X XX  XXXXXXXXXXXXXXX-X-XXXXXXXXXXXXXXX* 

OFFICER  #18 


A.  JOB  FACTORS: 

1.  Accomplished  all  assignments. 

2.  Efficient  in  the  use  of  time  and  other  resources. 

3.  On-the-job  communications  very  well  done. 

B.  PERSONAL  FACTORS: 

No  distinguishing  observations,  either  good  or  bad,  are  noted. 

C.  An  OER  rating  of  "1"  was  given. 

DECISION  #18 

How  appropriate  is  the  OER  rating  that  was  given?  (circle  one  number) 

1 2 3 4 5 

Definitely  NOT  Probably  NOT  Cannct  Probably  Definitely 

Appropriate  Appropriate  Determine  Appropriate  Appropriate 


no 
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PART  III 


EXERCISE  INSTRUCTIONS 


In  this  section,  you  are  asked  to  give  your  reac- 
tion to  four  hypothetical  situations  wherein  you  are 
either  receiving  (as  a ratee)  or  awarding  (as  a rater) 
various  OER  ratings.  Assume  that  the  interpretation  of 
a "1"  or  a "3"  is  the  same  as  in  Part  II  of  this  exercise, 
i.e.  22%  of  rated  officers  receive  "Is,"  58%  receive  "2s," 
and  20%  receive  "3s."  Also  assume  that  there  is  agreement 
among  the  rater,  additional  rater,  and  the  reviewer  con- 
cerning the  rating  assigned. 


Ill 


A.  You  have  .just  received  an  PER  rating  of  "3."  For  each  of  the  possible 
reactions  listed  below,  circle  the  number  which  best  describes  your 
feeling  toward  each  reaction. 


1.  They  (my  boss,  my  chain  of  command,  the  ’•system”)  don’t  think 
I should  be  promoted. 


Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Net  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

2.  They  are  saying  that  most  other  people  performed  their  job 
better  than  I did  mine. 

1 

o 

. . 

4 

Definitely 
Not  My 
Reaction 

Probably  Unsure 

Not  My 

Reaction 

Probably 

My 

Reaction 

Definitely 

My 

Reaction 

3.  They  don't  think  I could  be 

effective 

in  higher  level 

jobs. 

1 

, .2 

4 

Definitely 
Not  My 
Reaction 

Probably 

Not  My 

Reaction 

Unsure 

Probably 

My 

Reaction 

Definitely 

My 

Reaction 

4.  They  are 

saying  I didn't  do 

my  job  very  well. 

1 

2 

. . .3 

4 

Definitely 
Not  My 
Reaction 

Probably 

Not  My 

Reaction 

Unsure 

Probably 

My 

Reaction 

Definitely 

My 

Reaction 

B.  You  have  .just  given  an  PER  rating  of  "1"  to  another  officer.  For 
each  of  the  possible  reasons  listed  below,  circle  the  number  which 
best  describes  your  feeling  toward  each  reason. 


1.  The  officer  was  an  outstanding  performer  during  this  period. 


1 

Definitely 
Not  My 
Reasoning 

Probably  Unsure 

Not  My 

Reasoning 

....4 

Probably 

My 

Reasoning 

Definitely 

My 

Reasoning 

2 . 1 think 

this  officer  should  be  promoted. 

1 

..A 

Definitely 
Not  My 
Reasoning 

Probably  Unsure 

Not  My 

Reasoning 

Probably 

My 

Reasoning 

Definitely 

My 

Reasoning 

3.  I think  this  officer  could  do  a very  effective  job  at 
higher  levels. 

1 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 

4.  This  officer  performed  his/her  job  better  than  most  people. 

1 

3 4 

5 

Definitely 

Probably 

Unsure  Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 
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C.  You  have  .just  received  an  PER  rating  of  ”1."  For  each  of  the 
possible  reactions  listed  below,  circle  the  number  which  best 
describes  your  feeling  toward  each  reaction. 


1.  They  (my  boss,  my  chain 

oi  command,  the 

"system")  think  X 

could  be 

very  effective 

at  higher  level 

jobs. 

i 

1 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

2 . They  are 

saying  that  I performed  my  job  better  than  most  people. 

1 

4 

^ 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

3.  They  are 

saying  that  I did  my  job  very  well. 

i 

2 

• • » - *4*  • » • « • 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

4.  They  think  I should  be  promoted. 

1 

.,..4 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reaction 

Reaction 

Reaction 

Reaction 

You  have  .just  given  an  PER  rating  of  "3"  to  another  officer.  For 
each  of  the  possible  reasons  listed  below,  circle  the  number  which 
best  describes  your  feeling  toward  each  reason. 


1.  The  officer  did  not  perform  well  during  this  period. 


1 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 

2.  I think 

this  officer  should  not  be  promoted. 

1 

5 

Definitely 

Probably 

Unsure  Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 

3.  I think 

this  officer  could 

not  do  an 

effective  job  ai 

higher  levels 

1 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 

4.  Most  other  people  performed  their  jobs  better  than  this  officer. 


1 

O 

5 

Definitely 

Probably 

Unsure 

Probably 

Definitely 

Not  My 

Not  My 

My 

My 

Reasoning 

Reasoning 

Reasoning 

Reasoning 

I.  RATEE  IDENTIFICATION  DAT  A ( Read  AFR  36- 10  carefully  before  filling  in  any  item) 


i.  name  (Last,  First,  Middle  Initial)  3 ssn  /Include  suffix) 


2.  ORGANIZATION,  COMMAND,  LOCATION,  PAS  CODE  «.  PATSC  5.  DAFSC 


6.  ACTIVE  DUTY  GRADE 


8 PERIOD  OF  REPORT 


FROM 

THRU 


9.  DAYS  OF  SUPERVISION 


7.  PERMANENT  GRADF 


10.  REASON  FOR  REPORT 


II.  JOB  DESCRIPTION  i.  out y title: 

2 KEY  DUTIES,  TASKS,  AND  RESPONSIBILITIES: 


III.  PERFORMANCE  FACTORS  not  observed  far  i below  i meets  above  i ' 

1 Specific  example  of  performance  required  not  relevant  standard  standard  standard  standard  standard 


1.  job  knowledge  (Depth,  currency,  breadth)  Q 


2.  judcment  and  decisions  (Consistent,  Q 

accurate,  effective)  w 


3.  plan  and  organize  work  (Timely,  creative)  Q 


4 management  of  resources  (Manpower 
and  material) 


s.  leadership  (Initiative,  accept  responsibility) 


6 adaptability  to  stress  (Stable,  flexible, 
dependable ) 


7.  oral  communication  (Clear,  concise, 
confident) 


AF  opr,,  707  PREVIOUS  EDITION  IS  obsolete. 


8.  written  communication  (Clear,  concise, 
organized) 

0 

1 1 

1 1 

1 1 

1 1 

i i 

9.  professional  QUALITIES  {Attitude,  dress , 
cooperation,  bearing) 

0 

1 

! to  human  relations  (F.quat  opportunity 
participation,  sensitnity) 

1 

1_ 

OFFICER  EFFECTIVENESS  REPORT 


IV. 


RECOMMENDED  ASSIGNMENT  INFORMATION 


1.  STRONGEST  QUALIFICATION: 

2.  suggested  job  assignment  (Include  AFSC): 


V.  EVALUATION  OF  POTENTIAL 


V ORGANIZATION  LEVEL: 
4.  TIMING: 


TOP  BLOCK  CONTROLLED 

Evaluate  the  ratee 's  capability  relative  to  that  of  officers  in  the  same  grade  in  the  group  being 
evaluated,  for  expanded /mor*  diverse  responsibility.  Indicate  your  rating  by  placing  an  "X" 
in  the  designated  portion  of  the  appropriate  block. 


RATER  rAaDt°Nb  REVR  RATER  r*tER  «EVR 

Lowest  £ 


AOON 

RATER 


ADDN 

RATER 


IF  REPORT  IS  ABBREVIATED, 
‘•X*’  APPROPRIATE  BLOCK 
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Appendix  D 


Calculation  of  F-Test  Values 


The  F-test  values  used  to  compare  regression  models  in  this 


study  were  calculated  using  the  following  formula: 


_ [SS&  - SSej]  / [(p-1)  (K  + 1 )] 


Cj?1SSej^  / tn-P  <k+i)] 


where  SSg  is  the  residual  sum  of  squares  derived  by  regressing  all 


compared  groups  of  decisions  together,  SS  . is  the  residual  sum  of 

eJ 


squares  for  the  jth  group  of  evaluations,  p is  the  number  of  groups 


being  compared  (number  of  subsets  of  data  in  the  regression),  k is  the 


number  of  predictor  variables  (two,  in  all  cases  for  this  study)  and  n 


is  the  total  number  of  decisions  in  all  groups  being  compared. 


The  null  hypothesis  being  tested  is 


Hn:  P = P = P = . . . = P , where  P.  - ; 


1 2 3 


The  alternate  hypothesis  is 


,H  : J3  / P , for  at  least  one  i,  j pair. 

i j 


The  null  hypothesis  is  rejected  if 


F0  > F<*’  ^p_1)  <k+1^»  [n-p(k+l)3 


where  0 » 


0.05  in  all  comparisons  made  for  this  study, 
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This  research  was  conducted  to  examine  Air  Force  officers'  perceptions 
both  individually  and  in  groups,  of  what  is  involved 'in  a rating  of  potential. 

In  order  to  determine  these  perceptions,  a judgment  modeling,  or  policy  cap- 
turing research  approach  was  used.  A decision-making  t ercise  was  administer- 
ed to  381  active  duty  USAF  officers  attending  Squadron  Officers'  School,  Air 
Command  and  Staff  College,  and  Air  War  College.--  In  this  exercise,  officers 
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,ar>  asked  to  judge  the  appropriateness  of  OER  ratings  assigned  to  18 
hypothetical  officers,  based  on  different  levels  of  two  factors*  (1)  the 
officer* o past  performance,  as  illustrated  by  three  indicators  e.nd  (2)  the 
\ ^ of floor's  potential,  as  illustrated  by  three  indicators  other  than  past 
''performance.  Data  collected  from  the  exercise  were  then  analyzed  to  test 
various  hypotheses  concerning  the  relative  influence  of  performance  and 
(otWr)  potential  factors  on  officers'  judgments  of  OER  ratings. 

*“hia  results  of  the  research  indicate  that  Air  Force  officers  do  not 
view  tlw  OSR  rating  as  being  solely  determined  by  past  performance,  al- 
though the  performance  factor  was  weighted  most  heavily.  Analysis  indica- 
ted that  officers  used  different  judgment  policies  when  considering  "good" 
and  "hid”  OER  ratings,  and  that  they  were  generally  internally  consistent 
in  their  judgment  policies.  Additionally,  several  different  analyses  in- 
dicated that  the  Air  War  College  students  used  judgment  policies  that  were 
significantly  different  than  the  other  groups*  specifically,  AVC  students 
generally  placed  greater  emphasis  on  potential  than  the  other  PME  groups. 
Furthermore , the  AWC  group  displayed  both  the  lowest  internal  judgment 
consistency  and  the  lowest  group  R^  values  (an  indication  of  the  degree 
of  decision-making  homogeneity  within  the  group). 
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